Control apparatus, optical apparatus, control method, and non-transitory computer-readable storage medium

ABSTRACT

A control apparatus includes a signal readout unit  15  which reads out a frame image obtained from an image pickup device while the frame image is divided into a plurality of different regions, an image information calculating unit  16  which calculates image information based on an image signal of each of the plurality of different regions obtained from the signal readout unit, and an adjusting unit  17  which determines a target adjustment value of an image pickup unit including an image pickup optical system and the image pickup device based on the image information during capturing the frame image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an optical apparatus such as a digitalcamera and a digital video.

2. Description of the Related Art

Japanese Patent Laid-open No. 2002-27326 discloses a digital camerawhich reads out an image signal obtained from an identical pixel in aframe a plurality of times to perform a vibration correction (imagestabilization) based on a motion vector obtained from each image signal.

However, in the digital camera disclosed in Japanese Patent Laid-openNo. 2002-27326, a signal of the identical pixel in a frame is read out aplurality of times, and accordingly a noise is superimposed on an imageobtained in an entire frame.

SUMMARY OF THE INVENTION

The present invention provides a control apparatus, an image pickupapparatus, a control method, and a non-transitory computer-readablestorage medium which are capable of reducing a noise superimposed on animage when controlling an image pickup unit based on a plurality ofimage signals in a frame.

A control apparatus as one aspect of the present invention includes asignal readout unit configured to read out a frame image obtained froman image pickup device while the frame image is divided into a pluralityof different regions, an image information calculating unit configuredto calculate image information based on an image signal of each of theplurality of different regions obtained from the signal readout unit,and an adjusting unit configured to determine a target adjustment valueof an image pickup unit including an image pickup optical system and theimage pickup device based on the image information during capturing theframe image.

An optical apparatus as another aspect of the present invention includesan image pickup device, a signal readout unit configured to read out aframe image obtained from the image pickup device while the frame imageis divided into a plurality of different regions, an image informationcalculating unit configured to calculate image information based on animage signal of each of the plurality of different regions obtained fromthe signal readout unit, and an adjusting unit configured to determine atarget adjustment value of an image pickup unit including an imagepickup optical system and the image pickup device based on the imageinformation during capturing the frame image.

A control method as another aspect of the present invention includes thesteps of reading out a frame image obtained from an image pickup devicewhile the frame image is divided into a plurality of different regions,calculating image information based on an image signal of each of theplurality of different regions, and determining a target adjustmentvalue of an image pickup unit based on the image information duringcapturing the frame image.

A non-transitory computer-readable storage medium storing a programwhich causes a computer to execute a process including the steps ofreading out a frame image obtained from an image pickup device while theframe image is divided into a plurality of different regions,calculating image information based on an image signal of each of theplurality of different regions, and determining a target adjustmentvalue of an image pickup unit based on the image information duringcapturing the frame image.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image pickup apparatus in a firstembodiment.

FIG. 2 is an explanatory diagram of an operation of a signal readoutunit in the first embodiment.

FIGS. 3A to 3E are explanatory diagrams of a method of calculating animage blur correction amount in the first embodiment.

FIGS. 4A and 4B are explanatory diagrams of an effect of reducing areadout time in the first embodiment.

FIGS. 5A and 5B are explanatory diagrams of another method ofcalculating the image blur correction amount in the first embodiment.

FIGS. 6A to 6C are explanatory diagrams of another method of calculatingthe image blur correction amount in the first embodiment.

FIGS. 7A to 7G are flowcharts of an image blur correction in the firstembodiment.

FIGS. 8A and 8B are explanatory diagrams of a focus detection structureand focusing in the first embodiment.

FIGS. 9A to 9D are explanatory diagrams of the focusing in the firstembodiment.

FIGS. 10A and 10B are respectively a flowchart of the focusing and anexplanatory diagram of a focus detection in the first embodiment.

FIG. 11 is an explanatory diagram of a signal readout in a secondembodiment.

FIGS. 12A to 12N are explanatory diagrams of a relationship of imageinformation extraction ranges in the second embodiment.

FIG. 13 is a flowchart of creating an image blur correction plot in thesecond embodiment.

FIGS. 14A and 14B are explanatory diagrams of an operation of a signalreadout unit for capturing a still image in a third embodiment.

FIGS. 15A and 15B are explanatory diagrams of another operation of thesignal readout unit for capturing the still image in the thirdembodiment.

FIGS. 16A and 16B are explanatory diagrams of another operation of thesignal readout unit for capturing the still image in the thirdembodiment.

FIG. 17 is a flowchart of setting a readout order in the thirdembodiment.

FIGS. 18A and 18B are explanatory diagrams of an operation of a signalreadout unit for capturing a still image in a fourth embodiment.

FIGS. 19A and 19B are flowcharts of an image blur correction forcapturing the still image in the fourth embodiment.

FIGS. 20A and 20B are explanatory diagrams of another operation of thesignal readout unit for capturing the still image in the fourthembodiment.

FIGS. 21A and 21B are explanatory diagrams of an operation of a signalreadout unit for capturing a still image in a fifth embodiment.

FIGS. 22A to 22C are explanatory diagrams of a signal readout and a lenscontrol in a sixth embodiment.

FIGS. 23A and 23B are respectively an explanatory diagram and aflowchart of a control of a movable aperture stop during a continuousimage capturing and a flowchart of illustrating in each embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings.

First Embodiment

First of all, referring to FIG. 1, a configuration of an opticalapparatus (image pickup apparatus) in a first embodiment of the presentinvention will be described. FIG. 1 is a block diagram of an imagepickup apparatus 100 in this embodiment.

In FIG. 1, reference numeral 11 denotes an optical axis (photographingoptical axis), and reference numeral 12 denotes an image pickup opticalsystem. The image pickup optical system 12 includes an image blurcorrection lens 12 a that is driven in a direction which is(approximately) orthogonal to the optical axis 11 (i.e., in a planeorthogonal to the optical axis) to correct an image blur, a focus lens12 b that is driven along the optical axis 11 to perform focusing, and amovable aperture stop 12 c that limits a photographing light beam.Reference numeral 13 a denotes a lens barrel (lens apparatus) includingthe optical pickup optical system 12, and reference numeral 13 b denotesa camera body (image pickup apparatus body) including an image pickupdevice 14 (image pickup element or image sensor). In this embodiment,the image pickup apparatus 100 includes the camera body 13 b includingthe image pickup device 14 and the lens barrel 13 a removably attachedto the camera body 13 b. However, this embodiment is not limitedthereto, and it can be applied also to an image pickup apparatusincluding the image pickup apparatus body and the lens apparatusintegrated with each other.

Reference numeral 14 denotes the image pickup device. An image pickupunit is constituted by the image pickup optical system 12 and the imagepickup device 14. Reference numeral 14 a denotes a mechanical shutter(light shielding unit) that, as appropriate, shields an object lightbeam which is incident on the image pickup device 14 from the imagepickup optical system 12. Reference numeral 15 denotes a signal readoutunit that reads out a signal (image signal) which is output from theimage pickup device 14. Reference numeral 16 denotes an imageinformation calculating unit that calculates, as a signal forcontrolling the image blur correction lens 12 a or the focus lens 12 b,the image signal read from the image pickup device 14 based on an inputsignal from the signal readout unit 15. The image informationcalculating unit 16 includes an image information distribution detectingunit 16 a, a feature point coordinate calculating unit 16 b, a featurepoint coordinate estimating unit 16 c, an image information comparingunit 16 d, an image information complementing unit 16 e, and a temporarymemory unit 16 f.

Reference numeral 17 denotes an adjusting unit that outputs a targetadjustment value for at least one of the image blur correction lens 12a, the focus lens 12 b, and the movable aperture stop 12 c based on anoutput signal from the image information calculating unit 16. Referencenumerals 18 a, 18 b, and 18 c are drive units that drive the image blurcorrection lens 12 a, the focus lens 12 b, and the movable aperture stop12 c, respectively. Details of the signal output unit 15, the imageinformation calculating unit 16, and the adjusting unit 17 will bedescribed below. Reference numeral 19 denotes an image processing unitthat performs signal processing such as forming a luminance signal or acolor signal based on the signal read from the image pickup device 14and that generates a viewing image by performing gamma correction andcompression processing. A signal output from the adjusting unit 17 isalso input to the image processing unit 19, and the image processingunit 19 performs a brightness adjustment of an image based on the inputsignal from the adjusting unit 17.

Reference numeral 110 denotes a memory unit that stores a signal fromthe image processing unit 19. Reference numeral 111 denotes a CPU(control unit, or processor) that controls the signal readout unit 15and the image information calculating unit 16. Reference numeral 112denotes an operating unit that receives an instruction to capture animage by a user and that outputs, to the CPU 111, a signal based on theinstruction to capture the image. The image blur correction lens 12 a isdriven in the direction which is approximately orthogonal to the opticalaxis 11 (in the plane which is approximately orthogonal to the opticalaxis), and accordingly it can correct an image blur that is caused by ahand shake of the user and that occurs in an imaging plane of the imagepickup device 14. However, this embodiment is not limited to thereto,and instead of driving the image blur correction lens 12 a, the imagepickup device 14 may be driven in the direction which is approximatelyorthogonal to the optical axis 11 to correct the image blur that iscaused by the hand shake and that occurs in the imaging plane. Thisembodiment creates, in a frame, the target adjustment value of the imagepickup unit such as the image blur correction lens 12 a, the focus lens12 b, and the movable aperture stop 12 c based on a plurality of imagesignals in the frame. First, in this embodiment, a case in which animage blur correction is performed by controlling the image blurcorrection lens 12 a will be described.

Subsequently, referring to FIG. 2, the operation of the signal readoutunit 15 will be described. FIG. 2 is an explanatory diagram of theoperation of the signal readout unit 15. As illustrated in FIG. 2, aplurality of image blur correction plots 21 a′ to 24 e′ are obtainedfrom image signals obtained based on an instruction of the signalreadout unit 15 in frames 21, 22, 23, and 24 each divided into aplurality of regions “a” to “e” in a height direction of an imagingplane. A horizontal axis in FIG. 2 indicates a time. An accumulationtime of one frame is indicated as a time period 26, and for example itis 1/30. A vertical axis in FIG. 2 indicates a position in a heightdirection (imaging-plane height) of an image obtained in each frame andan image blur correction amount that occurs on a surface (imaging plane)of an image pickup element. FIG. 2 is an explanatory diagram of arolling readout in the image pickup element with a so-called CMOSstructure.

The image signals in the regions 21 a to 21 e are read out after thepassage of the accumulation time (time period 26). According to thisembodiment, since the signal readout is not performed during theaccumulation of the image signal differently from a conventionaltechnology, an amount of a superimposed noise is small. As illustratedin FIG. 2, a time delay occurs for the readout of each image signal, andthe image signal which is sequentially read can be used as effectiveimage information by using the time delay. In other words, the imageinformation calculating unit 16 obtains an image shift amount (imageblur amount) on the imaging plane of the image pickup device 14 thatoccurs during the time delay by comparing the image signals of each ofthe regions “a” to “e”.

Specifically, the image information calculating unit 16 obtains a firstfeature point coordinate based on the image signal in an imageinformation extraction range included in the region 21 a, and it obtainsa second feature point coordinate based on the image signal in an imageinformation extraction range included in the next region 21 b. Then, itobtains an image blur correction plot 21 b′ based on each coordinate. Amethod of calculating the image blur correction plot will be describedbelow. By repeating this operation, image blur correction plots 21 a′ to24 e′ are obtained. With respect to a region in which a feature point isnot included, the image information complementing unit 16 e obtains animage blur correction plot by performing prediction processing. Forexample, if there is not the image information extraction range in theregion 21 d, i.e., any feature point is not included in the region 21 d,the image signal (illustrated by a dashed line) obtained at that time isnot used. Black plots 21 d, 22 d′, 23 d′, and 24 d′ at that time arepredicted by the image information complementing unit 16 e provided inthe image information calculating unit 16 by using the image blurcorrection plots 21 a′ to 24 e′ and an adaptive filter or the like. Asdescribed above, the image blur correction plot in a region where anyfeature point is not obtained can be complemented.

The adjusting unit 17 outputs a target adjustment value of the imageblur correction lens 12 a depending on an image blur correction waveform25 obtained by the image blur correction plots 21 a′ to 24 e′, a focallength, object distance information, an optical sensitivity of the imageblur correction lens 12 a, an attitude of the image pickup apparatus100, and a situation of panning. The drive unit 18 a drives the imageblur correction lens 12 a during capturing a frame image based on thesignal from the adjusting unit 17 to decrease (reduce) the image blur.In the conventional technology, a blur coordinate is obtained bycomparing image signals in a common region between different frames. Forexample, a change of the blur is obtained by comparing the image signalsin the region 21 a and the region 22 a after all the signal readouts forthe frames 21 and 22 are completed, and it drives the image blurcorrection unit 12 a depending on the result or changes a position atwhich the image is cut out to decrease the image blur. On the otherhand, in this embodiment, a more accurate image blur correction can beperformed since a number of image blur correction plots are obtained inone frame, and also the superimposed noise can be effectively reducedsince the image signal is not read during the accumulation of the imagesignal.

Hereinafter, a method of obtaining an image blur correction amount byusing different image signals in a frame will be described. FIGS. 3A to3E are explanatory diagrams of a method of calculating the image blurcorrection amount. An image 31 in FIG. 3A is an object image obtained bythe readout sufficiently faster than the signal readout of each of theframes 21 to 24 illustrated in FIG. 2. The improvement of the readoutspeed will be described below. In FIG. 3A, a plurality of feature points33 (a symbol is allocated only to a representative point in FIG. 3A),for example like neon signs, exist at the background of an object (mainobject 32), and these are the image information extraction range. A handshake is distributed to be approximately uniform over an entire image.Accordingly, image blur correction amounts 34 indicate approximately thesame value in regions 35 a to 35 d illustrated in FIG. 3A. As describedreferring to FIG. 2, the time delay occurs in the image signals (forexample, image blur correction plots 21 a′ to 21 d′) obtained from theregions 35 a and 35 d with increasing the readout time, a difference ofthe respective image blur correction amounts occurs. In this embodiment,this phenomenon is used, and feature points that are different from eachother and that are distributed in for example the regions 35 a and 35 bin the same frame are compared to obtain an image blur amount.

FIG. 3B is an explanatory diagram of a method of obtaining the imageblur correction amount based on the comparison of the feature pointsdifferent from each other. First, the image information distributiondetecting unit 16 a provided in the image information calculating unit16 selects, as an image information extraction range, a feature pointwhich is suitable for a hand shake detection in an entire image to becaptured, and it obtains the distribution (relative position between thefeature points). The selection of the feature point is performed byobtaining a motion vector with the comparison between frames with thesame feature point for example, and then by selecting feature pointswhose directions are aligned (i.e., approximately the same direction).This is a known technology, and accordingly detail descriptions areomitted. Similarly to FIG. 3A, FIG. 3B illustrates feature points 33 a,33 b, and 33 d for reading the image blur correction amounts in theimage obtained with the sufficiently fast readout. Since the signalreadout is sufficiently fast, there is no influence of the hand shakedue to the readout time delay for each of the feature points 33 a, 33 b,and 33 d. Accordingly, a coordinate (xa,ya) of the feature point 33 a, acoordinate (xb,yb) of the feature point 33 b, and a coordinate (xd,yd)of the feature point 33 d can be obtained accurately. Thus, the relativeposition between the feature points 33 a and 33 b can be obtained.

FIG. 3C illustrates, similarly to FIG. 2, a situation in which only theimage signal in the region 35 a is read (a situation in which thereadout of an entire frame is not completed) when a slow readout isperformed. The feature point coordinate calculating unit 16 b providedin the image information calculating unit 16, for example, obtains thecoordinate (xa,ya) of the feature point 33 a in this time, and it setsthis coordinate as an initial value. The feature point coordinateestimating unit 16 c provided in the image information calculating unit16 estimates the coordinate (xb,yb) of the feature point 33 b at thattime based on the obtained coordinate of the feature point 33 a and therelative position between the feature points of FIG. 3B. In FIG. 3D, thefeature point coordinate calculating unit 16 b obtains a coordinate(xb′,yb′) of a feature point 33 b′ at that time when the readout iscompleted up to the image signal of the region 35 b. The readout of theregion 35 b has a time delay with respect to the readout of the region35 a. Accordingly, due to the influence of the hand shake during thetime, the coordinate (xb,yb) of the feature point 33 b estimated in FIG.3C is different from the coordinate (xb′,yb′) of the feature point 33 b′obtained in FIG. 3D. The image information comparing unit 16 d providedin the image information calculating unit 16 obtains the differencebetween both the coordinates to obtain an image blur correction amount34 b between the regions 35 a and 35 b considering the time delay.

The image information calculating unit 16 performs the calculationdescribed above while comparing the regions 35 a to 35 d in one frame,and it performs the same calculation for a next frame. Then, it sets forexample an image blur correction plot 21 a′ of the region 21 a in FIG. 2as an initial position, and obtains an image blur correction waveform 25based on the image blur correction plots between the respective featurepoints after that. However, in the method described above, it isnecessary to previously obtain the relative position with respect toeach of the feature points precisely. Accordingly, it is preferred thatthe time difference in reading the image signals is decreased as much aspossible to reduce the influence of the hand shake during the readouttime period. FIG. 3E is an explanatory diagram of the method.

In FIG. 3E, the image information extraction ranges (feature points 33a, 33 b, and 33 d) which are used to detect the hand shake in the image31 are selected from image information which has been obtained. As afeature point, a point having a large luminance difference (highcontrast) such as a neon sign is selected in the image 31. The imageinformation distribution detecting unit 16 a controls the signal readoutunit 15 so that these selected feature points are read out withpriority. The signal readout unit 15 reads only minimum regions 36 a, 36b, and 36 d including the feature points 33 a, 33 b, and 33 d,respectively, and it does not read other regions (i.e., performs adecimating readout or thinning readout). Alternatively, the signalreadout unit 15 reads a remaining region after reading the regions 36 a,36 b, and 36 d (i.e., changes the order to read the regions).

FIGS. 4A and 4B are explanatory diagrams of a reduction effect of thereadout time, and a horizontal axis and a vertical axis indicate a timeand an imaging-plane height, respectively. Referring to FIGS. 4A and 4B,the decimating readout and the change of the order of the readout,respectively, will be described. In FIG. 4A, only the regions 36 a, 36b, and 36 d in the image 31 are read out (regions including the selectedfeature points are read with priority). At this time, a time differencebetween image signals 41 a and 41 b and a time difference between imagesignals 41 a and 41 d are T1 and T2, respectively. Thus, compared with atime difference T3 between the regions 36 a and 36 b which is obtainedwhen the readout is performed in order from a line at the highestimaging plane, the readout time can be significantly reduced.

Similarly, in FIG. 4B, first, an image signal 41 e is read out from theregion 36 e, and then the regions 36 a, 36 b, and 36 d are read out(regions including the selected feature points are read with priority).After the readout in each region is completed, regions 36 f, 36 g, and36 h (image signals 41 f, 41 g, and 41 h) are read out. Also in thiscase, respective time differences of for the regions 36 a, 36 b, and 36d can be set to T1 and T2.

In FIG. 4A, a total readout time can be reduced since only the regions36 a, 36 b, and 36 d in the image 31 is read out. On the other hand, inFIG. 4B, a total readout time is long since the regions 36 e, 36 f, 36g, and 36 h are also read out. In FIG. 4B, however, the entire region isread out and accordingly a viewing image can be created by using theimage signals. Alternatively, as a middle situation between FIGS. 4A and4B, the readout order can be changed while a decimating is appropriatelyperformed. By performing this readout, the readout time difference ofthe regions including the feature points can be reduced and the relativeposition of each of the feature points can be obtained with highaccuracy.

As another method of obtaining the relative position of each of thefeature points with high accuracy, there is a method of obtaining therelative position of each of the feature points a plurality of times andthen averaging them to reduce the influence of the hand shake. In thismethod, it is not necessary to improve the readout speed. FIGS. 5A and5B are explanatory diagrams of another method of calculating the imageblur correction amount in this embodiment. In a plurality of frames 51,52, and 53 illustrated in FIG. 5A, the relative position of each of thefeature points is obtained in each frame, and the relative positions areaveraged between frames. In other words, coordinates of the same featurepoint between the plurality of read frames are averaged. In FIG. 5A,based on regions 51 a to 53 d read from each of the frames 51, 52, and53, coordinates (xa1,ya1) to (xd3,yd3) of the feature points 33 a, 33 b,and 33 d illustrated in FIG. 3E can be obtained. It is possible toobtain the relative position between the feature points based on thecoordinates (xa,ya), (xb,yb), and (xd,yd) averaged for each of thefeature points.

As another method of obtaining the relative position of each of thefeature points with high accuracy, there is a method of using a blurdetection result between frames. If a blur amount between the frames isknown, a blur in a frame can be estimated. Accordingly, a relativeposition error of each feature point caused by the blur is corrected byusing the estimated blur amount. In other words, based on a change ofthe coordinate of the same feature point between the plurality of readframes, a distribution of each feature point detected in one frame iscorrected. An image blur correction waveform 54 is obtained based onimage blur correction plots 51 g′ to 53 g′ obtained from regions 51 g,52 g (combined region of regions 51 a to 52 d), and 53 g in theplurality of frames 51, 52, and 53 illustrated in FIG. 5B. From theframe 52, image blur correction plots 52 a′ and 52 b′ are obtained basedon each of the regions 51 a to 51 d in the frame. A relative positionrelation between the image blur correction plots 52 a′ and 52 b′ can bedetected by application to the image blur correction waveform 54.

As described above, the relative position of each feature point in aframe can be precisely obtained by using the decimating readout in FIG.4A, the change of the readout order in FIG. 4B, the averaging in FIG.5A, and the blur between frames in FIG. 5B, in cooperation with thesignal readout unit 15 and the image information calculating unit 16.Then, the image blur correction amount between respective feature pointsis obtained by the method described referring to FIGS. 3A to 3D by usingthe image information comparing unit 16 d provided in the imageinformation calculating unit 16. Finally, the adjusting unit 17 obtainsthe image blur correction amount as a target image blur correctionvalue. Then, the drive unit 18 a drives the image pickup unit (imageblur correction lens 12 a or image pickup device 14) in a directionapproximately orthogonal to the optical axis 11 based on the imageobtained in a frame during the signal readout for the frame. In thisembodiment, the method of performing the blur correction is not limitedto the method of driving the image blur correction lens 12 a or theimage pickup device 14 in the direction approximately orthogonal to theoptical axis 11 described above. For example, other methods such as atilt drive of the blur correction lens 12 a and a drive depending on animage blur correction amount for the entire image pickup unit(stabilization in an absolute space) can also be applied. As describedabove, in this embodiment, the image blur correction can be performedbased on the plurality of image blur correction amounts obtained in oneframe. Accordingly, it is possible to perform the image blur correctionwith high response. Since this embodiment does not read a signal of thesame pixel in one frame a plurality of times differently from theconventional technology, the superimposition of a noise on an image canbe reduced. As above, the image blur correction processing in thesituation where the feature points to be targeted are distributed overan entire screen (image) like a hand-shake situation (FIGS. 3A to 3E) isdescribed.

Subsequently, referring to FIGS. 6A to 6C, a case in which the mainobject is a moving object and an image is to be captured while followingthe moving object will be described. FIGS. 6A to 6C are explanatorydiagrams of another method of calculating the image blur correctionamount in this embodiment. FIG. 6A illustrates an image 61 obtained in acase where a main object 62 is a moving object and a so-called panning(follow shot) which captures an image while following the main object 62is performed. In this case, feature points 63 are set at the main object62, but the feature points 63 are densely located and thus they are notdispersed differently from the case of FIGS. 3A to 3E. Accordingly, itis difficult to obtain a stable image blur correction waveform. In orderto obtain image signals in a region including each feature point whilebeing dispersed, the readout order of the image signals is changed.

In FIG. 6A, the signal readout unit 15 reads regions 65 a to 65 e whichinclude the feature points 63 and regions 66 a to 66 e which do notinclude the feature point 63 based on coordinates of the feature pointsselected by the image information distribution detecting unit 16 a. Thenumber of the regions including the feature point 63 is set depending onimage signals needed in one frame. For example, if five plots arerequired as the image blur correction plots in order to obtain the imageblur correction waveform in one frame, the image informationdistribution detecting unit 16 a ensures five regions as regionsincluding feature points. In addition, it ensures regions having thesame number which do not include the feature points. Then, the imageinformation distribution detecting unit 16 a controls the signal readoutunit 15 so as to alternately read the region which includes the featurepoint 63 and the region which does not include the feature point 63, andit obtains an image blur correction amount at equal intervals(approximately constant intervals) in one frame.

FIG. 6C is an explanatory diagram of image signals in each region and animage blur correction waveform obtained based on the image signals whilethe readout order is changed as described above. For comparison, FIG. 6Billustrates image signals in each region and an image blur correctionwaveform obtained based on the image signals while the readout order isnot changed. In frames 67, 69, 611, and 613, the image signals are readin order from the region in which the imaging-plane height is higher.Accordingly, image signals in gray regions 67 a to 67 e (69 a to 69 e,611 a to 611 e, and 613 a to 613 e) including feature points as imageinformation extraction ranges are adjacently read out, and image blurcorrection plots are obtained to be concentrated for each frame. Asdescribed above, the image blur correction waveform of intervals (68 ato 68 e, 610 a to 610 e, 612 a to 612 e, and 614 a to 614 e) in whichthe image blur correction plot cannot be obtained is approximated byusing straight lines 615 a to 615 c, and accordingly the accuracy of theimage blur correction is decreased.

On the other hand, in this embodiment, as illustrated in FIG. 6C, theimage signals in the gray regions 67 a to 67 e which include the featurepoints and the image signals in the white regions 68 a to 68 e, 610 a to610 e, 612 a to 612 e, and 614 a to 614 e which do not include anyfeature points are read alternately. As a result, since the image blurcorrection plots are distributed uniformly, and accordingly ahighly-accurate image blur correction waveform 617 is obtained. Grayplots 616 a to 616 c are image blur correction plots which arepredictively complemented by the image information complementing unit 16e by using a known technology such as a Kalman filter. As describedabove, the image blur correction plots obtained from the image signalsare uniformly distributed at narrow intervals, and accordingly theprediction accuracy can also be improved. By driving the image blurcorrection lens 12 a based on the image blur correction waveform 617obtained as described above, it is possible to perform the image blurcorrection following the moving object.

FIG. 7A is a flowchart of the image blur correction in this embodiment.In FIG. 7A, for a simple explanation, elements which are not directlyrelevant in this embodiment are omitted. This flow starts when forexample a user performs a photographing preparation operation such as ahalf-press of a release button provided on a camera and the operatingunit 112 outputs an instruction for the photographing preparation to theCPU 111. This flow is finished when the instruction for thephotographing preparation is canceled. Each step in FIG. 7A is performedmainly the signal readout unit 15, the image information calculatingunit 16, the adjusting unit 17, the drive unit 18 a, or the imageprocessing unit 19 based on an instruction of the CPU 111.

First, at step S701, the image information distribution detecting unit16 a selects an image information extraction range (feature point) on ascreen (image) based on the image obtained by the image processing unit19 or the image information calculating unit 16. As described referringto FIG. 3E, points each having a large luminance difference (highcontrast) such as a neon sign in the image 31 are selected as featurepoints. It is preferred that the feature points are set uniformly in theimage (screen) upward and downward. This is because sampling intervalsof image signals corresponding to the feature points are unbalancedduring the image readout if the feature points are set to beconcentrated in one region (a predetermined region) in the screen. Ifthe feature points are set to be concentrated in the predeterminedregion, steps S703 and S704 below may be needed.

Subsequently, at step S702, the image information calculating unit 16(feature point coordinate calculating unit 16 b) detects a coordinate ofthe feature point as an image information extraction range. As describedreferring to FIGS. 4A, 4B, 5A, and 5B, this is because the high-speedreadout at which the influence of the hand shake does not occur and theblur amount correction is performed to detect a coordinate of thefeature point with high accuracy. Subsequently, at step S703, the imageinformation calculating unit 16 (image information distributiondetecting unit 16 a) sets the readout order for the signal readout unit15. If the feature points are uniformly distributed over the entirescreen, image signals are read in order from an upper region toward alower region in the screen. On the other hand, if the feature points areconcentrated in the predetermined region, as described referring to FIG.6C, the readout order is changed to set the sampling of the imagesignals to be approximately equal intervals.

Subsequently, at step S704, the signal readout unit 15 reads the imagesignal in each region in the readout order set at step S703.Subsequently, at step S705, as described referring to FIGS. 3A to 3E,the image information comparing unit 16 d compares the read imagesignals for each region. Then, the image information calculating unit 16generates the image blur correction plot based on the comparison result.Subsequently, at step S706, the adjusting unit 17 generates a targetadjustment value that is to be used for the image blur correction basedon the obtained image blur correction plot. Then, at step S707, thedrive unit 18 a drives the image blur correction lens 12 a based on thetarget adjustment value generated by the adjusting unit 17.

Subsequently, at step S708, the signal readout unit 15 reads the imagesignal in the next region. Then, at step S709, the CPU 111 (or thesignal readout unit 15) determines whether image signals in regions (allregions) needed to form an image in one frame are read out. If thenecessary regions are read out, the flow proceeds to step S710. On theother hand, if a region to be read in one frame remains, the flowreturns to step S704, and the readout of image signals and the imageblur correction (steps S704 to S708) are repeated.

At step S701, the process proceeds to the next frame. Subsequently, atstep S711, the image information calculating unit 16 determines whethera composition in the screen is changed. If the composition is notchanged (or a change amount of the composition is smaller than apredetermined amount), the flow returns to step S704, and image signalsin the next frame are read with the condition set at steps S701 to S703.On the other hand, if the composition is changed (or the change amountof the composition is larger than the predetermined amount), the flowreturns to step S701, and the setting of the feature point, thedetection of the coordinate of the feature point, and the setting of thereadout order are performed again.

FIG. 7B is a subroutine relating to step S701 in FIG. 7A (selection ofthe image information extraction range (i.e., selection of the featurepoint)). At step S7011, the image information distribution detectingunit 16 a starts scanning in a height direction (row direction) of thescreen for the image obtained by the image processing unit 19 or theimage information calculating unit 16. Subsequently, at step S7012, theimage information distribution detecting unit 16 a determines, withrespect to the scanned row, whether a luminance difference betweenpixels is larger than a predetermined value. If there is a pixel groupwhere the luminance difference is larger than the predetermined value,the flow proceeds to step S7013. On the other hand, there is no pixelgroup where the luminance difference is larger than the predeterminedvalue, the flow returns to step S7011, and scanning is to be performedfor the next row. Since the luminance difference in the row directioncannot be obtained based on the image signals for each row in thescreen, the luminance value in the row direction is obtained based on areadout result of image signals in a plurality of rows (for example 10rows).

Subsequently, at step S7013, the image information distributiondetecting unit 16 a determines whether a distance in the row directionis shorter than a predetermined distance. Specifically, if a roughdistance in the row direction between a previously-obtained pixel grouphaving a large luminance difference and a currently-obtained pixel grouphaving a large luminance difference is shorter than the predetermineddistance, the flow proceeds to step S7014. On the other hand, thedistance in the row direction is longer than the predetermined distance,step S7014 is skipped and the flow proceeds to step s7015. Due to theinfluence of the hand shake and the like, the distance between thefeature points can be roughly obtained. This is not a problem inaccuracy with which a distribution of the pixel group having theluminance difference in the screen is seen. Step S7014 is provided totransmit the necessity to change the signal readout order to increasethe readout time interval between two points, as illustrated in FIG. 6C,from the image information distribution detecting unit 16 a to thesignal readout unit 15 if the distance in the row direction for the twofeature points is short.

At step S7015, the image information distribution detecting unit 16 aselects the obtained pixel group as a feature point needed for the imageblur detection (i.e., determines the image information extractionrange). When a plurality of feature points are obtained by the samescanning, instead of using all the feature points, the number of theselected feature points is adjusted depending on the photographing time(signal accumulation time) for the frame. For example, many featurepoints are used if the readout time of the image signals is long and itis necessary to obtain the image signals with fine intervalstherebetween, and on the other hand, if not, the number of the featurepoints are decreased such that a calculation load per time is constant.Subsequently, at step S7016, the image information distributiondetecting unit 16 a determines whether the readout of the necessary rowsin one frame is completed. If the readout of the necessary rows is notcompleted, the flow returns to step S7011, and the scanning of the nextrow continues. On the other hand, if the readout of the necessary rowsis completed, this subroutine is finished, and the flow proceeds to stepS702.

FIG. 7C is a subroutine in the detection of the coordinate of thefeature point at step S702, and it illustrates an example of thedecimating readout and the readout order as described referring to FIGS.4A and 4B. First, at step S7021, the signal readout unit 15 sets onlythe minimum regions 36 a, 36 b, and 36 d including the feature points 33a, 33 b, and 33 d, respectively, illustrated in FIG. 3E as readoutregions. Subsequently, at step S7022, the signal readout unit 15 readsthe set readout region. Subsequently, at step S7023, the signal readoutunit 15 determines whether the readout of the set readout regions arecompleted. If the readout of the readout regions are completed, the flowproceeds to step S7024. On the other hand, if the readout of the readoutregions is not completed, the flow returns to step S7022, the readout ofimage signals (readout regions) continues.

If the readout of the readout regions set at step S7023 is completed,subsequent steps S7024 and S7025 are skipped in the method of thedecimating readout, and at step S7026, the image information calculatingunit 16 obtains the coordinate of the feature point. Then, thissubroutine is finished, and the flow proceeds to step S703. In themethod of changing the readout order, at step S7024, the signal readoutunit 15 reads a region which does not include the feature point.Subsequently, at step S7025, steps S7024 and S7025 are repeated tocontinue the readout of the image signals until the readout of theregion is completed. When the readout of the image signal in this regionis completed, the flow proceeds to step S7026, and the image informationcalculating unit 16 obtains the coordinate of the feature point in theset region. Then, this subroutine is finished and the flow proceeds tostep S703.

FIG. 7D is another example of the subroutine in the detection of thecoordinate of the feature point at step S702, and it illustrates anexample of averaging processing of the coordinate of the feature pointbetween frames as described referring to FIG. 5A. First, at step S7021′,the signal readout unit 15 reads image signals in order from the upperside in the screen. Subsequently, at step S7022′, the signal readoutunit 15 determines whether the readout of the image signals in thenecessary regions of the screen is completed. If the readout of theimage signals is completed, the flow proceeds to step S7023′. On theother hand, if the readout of the image signals is not completed, theflow returns to step S7021′ and the readout of image signals continues.

Subsequently, at step S7023′, the image information calculating unit 16obtains the coordinate of the feature point set at step S701 based onthe obtained image signal. Subsequently, at step S7024′, the processproceeds to the next frame. Subsequently, at step S7025′, the signalreadout unit 15 determines whether the readout of the image signals inpredetermined frames (for example, three frames in FIG. 5A) iscompleted. If the readout of the image signals in the predeterminedframes is not completed, the flow returns to step S7021′, and the signalreadout unit 15 reads an image signal in the next frame. On the otherhand, if the readout of the image signals in the predetermined frames iscompleted, the flow proceeds to step S7026′. At step S7026′, the imageinformation calculating unit 16 averages the same coordinates of thefeature points obtained in a plurality of frames to obtain thecoordinate of the feature point where the influence of the hand shakehas been reduced, and thus the accuracy of the coordinate of the featurepoint is improved. Then, this subroutine is finished and the flowproceeds to step S703.

FIG. 7E is another example of the subroutine in the detection of thecoordinate of the feature point at step S702, and it illustrates anexample of correcting the coordinate of the feature point of each regionin one frame by using the image blur correction waveform obtained fromthe same feature points between frames as described referring to FIG.5B. First, at step S7021″, the signal readout unit 15 reads imagesignals in order from the upper side in the screen. Subsequently, atstep S7022″, the signal readout unit 15 determines whether the readoutof the image signals in the necessary regions of the screen iscompleted. If the readout of the image signals is completed, the flowproceeds to step S7023″. On the other hand, if the readout of the imagesignals is not completed, the flow returns to step S7021″, and thesignal readout unit 15 continues the readout of the image signals.

At step S7023″, the image information calculating unit 16 obtains thecoordinate of the feature point set at step S701 based on the obtainedimage signal. Subsequently, at step S7024″, the process proceeds to thenext frame. Subsequently, at step S7025″, the signal readout unit 15determines whether the readout of the image signals in predeterminedframes (for example, three frames in FIG. 5A) is completed. If thereadout of the image signals in the predetermined frames is notcompleted, the flow returns to step S7021″, and the signal output unit15 reads image signals in the next frame. On the other hand, if thereadout of the image signals is completed, the flow proceeds to stepS7026″.

At step S7026″, the image information calculating unit 16 obtains theimage blur correction waveform based on a change of a position of thesame feature point between frames. Subsequently, at step S7027″, theimage information complementing unit 16 e obtains an image blurcorrection amount at the timing of the readout of the feature point inone frame set at step S701. In this case, the image informationcomplementing unit 16 e obtains the image blur correction amount, basedon the image blur correction waveform calculated at step S7026″, byusing a linear prediction or a prediction of the adaptive filterdescribed above or an LPC. Subsequently, at step S7028″, the imageinformation calculating unit 16 corrects the coordinate of the featurepoint obtained at step S7023″ based on the image blur correction amountcalculated at step S7027″ (i.e., prediction result). As described above,the accuracy of the coordinate of the feature point is improved, andthen this subroutine is finished and the flow proceeds to step S703.

FIG. 7F is a subroutine in the setting of the readout order at stepS703. First, at step S7031, the image information calculating unit 16determines whether there is a flag indicating the change of the readoutorder at steps S7014 in the flowchart of FIG. 7B. If there is the flag,the flow proceeds to step S7032. On the other hand, if there is no flag,this subroutine is finished, and the flow proceeds to step S704. At stepS7032, as described referring to FIG. 6A, the image informationdistribution detecting unit 16 a sets the plurality of regions eachhaving the feature point, and it divides regions which do not have anyfeature points into regions with the same number as that of the regionshaving the feature points. Subsequently, at step S7033, as describedreferring to FIG. 6C, the image information distribution detecting unit16 a sets the readout so that the region which has the feature point andthe region which does not have the feature point are alternately readout. Accordingly, outputs of the image signals obtained from the regionshaving the feature points are distributed uniformly at approximatelyequal intervals during the output time period of the image signals inone frame. When sorting of the readout order is completed at step S7033,this subroutine is finished and the flow proceeds to step S704.

FIG. 7G is a subroutine in the creation of the image blur correctionplot at step S705. First, at step S7051, as described referring to FIGS.3A to 3E, the image information calculating unit 16 obtains thecoordinate of the feature point in the current region based on the imagesignal read in the current region, and also it estimates a coordinate ofthe feature point in a region to be subsequently read based on thefeature point distribution in FIG. 3B. For example, in FIG. 3C, it isassumed that the feature point coordinate calculating unit 16 b obtainsthe coordinate of the feature point 33 a based on the image signal inthe current region 35 a. In this case, the feature point coordinateestimating unit 16 c estimates the feature point in the region 35 b tobe subsequently read based on the relationship between the coordinatesof the feature points 33 a and 33 b in the feature point distribution(detected at step S702) in FIG. 3B. Then, the image informationcalculating unit 16 stores the estimated feature point in the temporarymemory unit 16 f.

Subsequently, at step S7052, the image information calculating unit 16draws (reads) the coordinate of the feature point estimated at previousstep S7051 from the temporary memory unit 16 f. Hereinafter, a method ofestimating the coordinate of the feature point 33 a in the first readoutregion as the region 35 a in a frame will be described. When the featurepoint in the first region in a frame is to be estimated, the coordinate33 d of the feature point in the last region read in the previous frame(for example, the region 35 d of FIG. 3B in a previous frame) is used.In other words, when previous step S7051 has passed, the last region inthe previous frame has been read and the coordinate of the last featurepoint is obtained by using the last region. Then, the coordinate of thefeature point of the first region in the next frame (for example, thecoordinate of the feature point 33 a in the region 35 a in the nextframe) can be estimated based on the coordinate of the last featurepoint and the feature point distribution of FIG. 3B.

Subsequently, at step S7053, the image information comparing unit 16 dcompares the coordinate of the feature point at the current timeobtained at step S7051 with the coordinate of the feature point locatedat the same position estimated at the previous readout time drawn atstep S7052 to acquire the image blur correction plot. When passingthrough steps S7051 to S7053 in the next cycle, the image informationcomparing unit 16 d compares the coordinates of the feature points 33 band 33 b′ (FIGS. 3C and 3D) estimated at step S7051 in the previouscycle and the current cycle to acquire the image blur correction plot.After the image blur correction plot is acquired at step S7053, thissubroutine is finished and the flow proceeds to step S706.

As described above, the image blur correction plot can be acquired basedon the different coordinates of the feature points in one frame, and itis possible to perform the image blur correction in this frame. Thedifferent coordinates of the feature points are for example the featurepoints 33 a and 33 b as describe referring to FIGS. 3A to 3E. Based onthe coordinate of the feature point 33 a, the coordinate of the featurepoint 33 b included in a next region at that time is estimated, and thenthe estimated coordinate of the feature point 33 b is compared with thecoordinate of the feature point 33 b included in the region subsequentlyread to acquire the image blur correction plot.

As described above, the example of performing the image blur correctionby using the plurality of feature points in one frame is describedreferring to FIGS. 2 to 7A-7G. Next, an example of performing focusing(focus adjustment) by using a plurality of focus state detection unitsin one frame will be described.

FIGS. 8A and 8B are explanatory diagrams of a focus detection structureand focusing in this embodiment, respectively. FIG. 8A is an enlargedview of some pixels that constitute the image pickup device 14. Eachpixel includes a common microlens 81 and two photoelectric conversionelements 82 a and 82 b provided under the microlens 81. Accordingly,light beams from the image pickup optical system 12 pass through therespective regions of the common microlens 81 different from each other,and they enter the photoelectric conversion elements 82 a and 82 b,respectively. In other words, the light beams passing through pupilregions of the image pickup optical system 12 different from each otherare incident on the photoelectric conversion elements 82 a and 82 b. Thephotoelectric conversion elements 82 a and 82 b in each pixel groupconstituted as described above are correlated with each other, andaccordingly it is possible to detect the focus state of the image pickupoptical system 12. This is a known technology as a focus state detectionmethod by a phase-difference detection method using an imaging-planepixel, and it can perform the focusing by drive control of the focuslens 12 b based on the focus state detected on the imaging plane. Withrespect to each pixel of the image pickup device 14, the pairs ofphotoelectric conversion elements described above are arranged on anentire surface, and accordingly the focus state can be detected at allpoints on a photographing screen (i.e., captured image). For example, asillustrated in FIG. 8B, a case in which an object 83 that swings in thewind in a direction indicated by an arrow 84 is to be photographed bythe image pickup apparatus 100 (camera) is considered.

FIGS. 9A to 9D are explanatory diagrams of the focusing in thisembodiment. FIG. 9A illustrates a photographing composition in the imagecapturing condition illustrated in FIG. 8B, and there is an object to befocused at the range-finding frames 93 a 1 to 93 d 2 for the main object83. These range-finding frames correspond to the image informationextraction ranges. As illustrated in FIG. 9A, the range-finding frames93 a 1 to 93 d 2 are included in the regions 92 a to 92 d, and there isno object to be focused at the range-finding frames 93 e 1 and 93 e 2 inthe region 92 e.

With respect to the photographing composition of FIG. 9A, FIG. 9Billustrates a situation in which the in-focus state is detected byreading each image signal in one frame and the drive control of thefocus lens 12 b is performed. In each of the frames 91, 92, 93, and 94divided into the plurality of regions “a” to “e” in the direction of theimaging-plane height, a plurality of focus correction plots 91 a′ to 94e′ are obtained based on an instruction of the signal readout unit 15.In FIG. 9B, a horizontal axis indicates a time, and an accumulation timefor one frame is a time period 96, and for example it is 1/30. In FIG.9B, a vertical axis indicates a position (imaging-plane height) in aheight direction of an image obtained in each frame, and it is a diagramof a rolling readout in an image pickup element having a so-called CMOSstructure. The vertical axis also indicates an amount (focus correctionamount) which is used to perform the focus correction.

Each of the image signals in the regions 91 a to 91 e is read after thepassage of the accumulation time period. Accordingly, thesuperimposition of the noise can be suppressed compared with aconventional technology which reads image signals in the middle of theaccumulation. While a time delay occurs in the readout of each imagesignal as illustrated in the drawing, a time change in the focus statecan be densely acquired by using this delay. In other words, the imageinformation calculating unit 16 obtains the focus state occurring duringthis time delay by using the range-finding frames (93 a 1 to 93 d 2)included in the image information of each of the regions 91 a to 91 e.Specifically, the image information calculating unit 16 detects anaverage focus state based on each of the image signals of therange-finding frames 93 a 1 and 93 a 2 included in the region 91 a, andsubsequently it detects an average focus state based on each of theimage signals of the range-finding frames 93 b 1 and 93 b 2 included inthe region 91 b. By repeating the process, the focus state at each timeis detected.

With respect to the region 91 e in which there is not any object to befocused, the image information complementing unit 16 e obtains the focusstate by prediction processing. For example, plots 91 e′, 92 e′, 93 e′,and 94 e′ indicating focus states in the region 91 e are predicted byusing an adaptive filter or the like based on the plots 91 a′ to 94 d′indicating focus states. As described above, the plot indicating thefocus state at the readout time in the region where there is no objectto be focused is complemented. The adjusting unit 17 outputs the targetadjustment value of the focus lens 12 b. This target adjustment value isdetermined depending on a focus correction waveform 95 obtained by thefocus correction plots 91 a′ to 94 e′ indicating the focus state, afocal length of the image pickup unit, object distance information, anoptical sensitivity of the focus lens 12 b, and a position and a panningsituation of the image pickup apparatus. The drive unit 18 b drives,based on a signal of the adjusting unit 17, the focus lens 12 b duringcapturing a frame image in which the signal readout is performed tocorrect a change of focus. As described above, in this embodiment, theimage blur correction plots indicating many focus states in one frameare obtained, and therefore it is possible to perform a dense focuscorrection. Furthermore, since each of the focus correction signalsobtained from each range-finding frame is obtained after the imagesignals are accumulated, the superimposition of the noise can beeffectively reduced.

FIG. 9C illustrates a state in which a proportion of the main object 83in a screen 91 is smaller than that of the composition in FIG. 9A. Inthis case, a region including the range-finding frames 93 a 1 to 93 e 2of the main object 83 (region where there is an object to be focused) issmaller than FIG. A. Accordingly, it is necessary to read the imagesignals of the image information extraction ranges in this regionuniformly in one frame. Therefore, the image information distributiondetecting unit 16 a selects the range-finding frames 93 a 1 to 93 e 2 asimage information extraction ranges (focus state detection frames).Then, it divides the region where there is no object to be focused intoregions 94 a to 94 e having the same number as that of the regions 92 ato 92 e including the range-finding frames where there is an object tobe focused. Then, based on the setting of the image informationdistribution detecting unit 16 a, the signal readout unit 15 alternatelyreads the region where there is the object to be focused and the regionwhere there is no object to be focused.

FIG. 9D illustrates image signals in each region and a focus correctionwaveform obtained by the image signals while the readout order ischanged as described above. In this embodiment, image signals in grayregions 917 a to 923 e including the range-finding regions 93 a 1 to 93e 2 selected by the image information distribution detecting unit 16 aand image signals in white regions 918 a to 924 e including theunselected range-finding frames (range-finding frames where there is noobject to be focused) are read alternately. In this case, since thefocus correction plots are distributed uniformly, and accordingly ahighly-accurate focus correction waveform 926 is obtained. Gray plots925 a to 925 c are focus correction plots which are predictivelycomplemented by the image information complementing unit 16 e by using aknown technology such as a Kalman filter, and the focus correction plotsobtained from the image signals are uniformly distributed at narrowintervals. Accordingly, the prediction accuracy can also be improved. Bydriving the focus lens 12 b based on the focus correction waveform 926obtained as described above, it is possible to perform the focuscorrection while finely following a moving object such as a flowerswinging in the wind and a person riding on a swing.

FIGS. 10A and 10B are a flowchart of the focusing and an explanatorydiagram of focus detection in this embodiment, respectively. FIG. 10A isthe flowchart of the focusing described above, and for a simpleexplanation, elements which are not directly relevant to this embodimentare omitted. This flow starts when for example a user performs aphotographing preparation operation such as a half-press of a releasebutton provided on a camera and the operating unit 112 outputs aninstruction for the photographing preparation to the CPU 111, and it isfinished when the instruction for the photographing preparation iscanceled.

First, at step S1001, the image information distribution detecting unit16 a selects a range-finding frame (focus detection frame) in an imageas an image information extraction range based on the image obtained bythe image processing unit 19 or the image information calculating unit16. The range-finding frame is selected, for example, by setting anobject closest to the camera in FIG. 9C and setting an image regioncapturing a main object as a range-finding frame used for focusing. Therange-finding frames are set uniformly to the image (screen) upward anddownward. This is because sampling intervals of image signalscorresponding to the range-finding frames are unbalanced during theimage readout if the range-finding frames are set to be concentrated inone region (a predetermined region) in the screen.

Subsequently, at step S1002, the signal readout unit 15 sets a readoutorder. If the range-finding frames set over the entire image aredistributed uniformly, the signal readout unit 15 reads image signals inorder from the upper side toward the lower side of the image. On theother hand, when the set range-finding frames are concentrated in apredetermined region, as described referring to FIG. 9D, the signalreadout unit 15 changes the readout order to set the sampling of theimage signals to be approximately the same intervals. Subsequently, atstep S1003, the signal readout unit 15 reads the image signals for eachregion in the readout order set at step S1002. Subsequently, at stepS1004, the image information calculating unit 16 detects a focus stateof the read image signals for each region, and it creates a focuscorrection plot based on an instruction of the signal readout unit 15.As described above, when the focus state detection is performed by usingthe phase difference, image signals of a plurality of pixels in therange-finding frame set in the region are used.

FIG. 10B is an explanatory diagram of the focus state detection by aphase-difference detection method. A plurality of pixels 1002 to 1009are provided in the range-finding frame 1001. The image informationcomparing unit 16 d compares a shift amount between an image signal 1010a that is formed by signals of photoelectric conversion elements 1002 ato 1009 a constituting the respective pixels and an image signal 1010 bthat is formed by signals of photoelectric conversion elements 1002 b to1009 b. If peaks 1011 a and 1011 b of the respective two image signals1010 a and 1010 b coincide with each other, the focus state is a stateof focusing on an object (in an in-focus state). On the other hand, ifthe peaks 1011 a and 1011 b are shifted from each other as indicated byan arrow 1012, a focus correction value is obtained depending on theshift amount. Since the plurality of range-finding frames are set in oneregion, focus correction values obtained in the respective range-findingframes are averaged to be set as a focus correction value in the region.The focus state detection is not limited to the method of using thephase difference, but it is possible to evaluate the focus state byusing a contrast value of an image. Also in this case, it is possible toperform the focus correction using a change of the contrast value in theset range-finding frame.

Subsequently, at step S1005 in FIG. 10A, the adjusting unit 17 creates atarget adjustment value for focus correction (i.e., target focuscorrection value) based on the focus correction plot created at stepS1004. Subsequently, at step S1006, the drive unit 18 b drives the focuslens 12 b based on the target adjustment value created by the adjustingunit 17 at step S1005. Subsequently, at step S1007, the signal readoutunit 15 reads image signals in the next region. Subsequently, at stepS1008, the signal readout unit 15 determines whether the image signalsin the necessary regions for the image formation in one frame are readout. If the readout of the image signals is completed, the flow proceedsto step S1009. On the other hand, if any region to be read in one frameremains, the flow returns to step S1003, and repeatedly image signalsare read out to perform the focus correction based on the signals.

Subsequently, at step S1009, the process proceeds to the next frame.Subsequently, at step S1010, the image processing unit 19 or the imageinformation calculating unit 16 determines whether a composition of theimage (screen) is changed. If the composition is not changed, the flowreturns to step S1003, and for the next frame, the signal readout unit15 reads image signals with the condition set at step S1001. On theother hand, if the composition is changed, the flow returns to stepS1001, and the image information distribution detecting unit 16 a setsthe range-finding frame and the readout order again.

As described above, the focus correction plot can be obtained from thedifferent range-finding frames in one frame and the focus correction canbe performed in this frame. The different range-finding frames are forexample the range-finding frames 93 a 1 to 93 d 2 as described referringto FIG. 9A. Thus, in this embodiment, the image pickup apparatus 100includes the image pickup device 14 and the signal readout unit 15 thatreads the image in one frame obtained from the image pickup device 14while dividing the image into a plurality of different regions (forexample, regions 21 a to 21 e). Furthermore, the image pickup apparatus100 includes the image information calculating unit 16 that calculatesimage information (blur correction plots 21 a′ to 21 e′, focuscorrection plots 91 a′ to 91 d′, and the like) based on image signals ineach of the different regions obtained from the signal readout unit 15.In addition, the image pickup apparatus 100 includes the adjusting unit17 that creates a target adjustment value of an image pickup unit duringcapturing the frame based on each of pieces of image informationcalculated by the image information calculating unit 16. Furthermore,the image pickup apparatus 100 includes the drive unit 18 (18 a, 18 b)that drives the image pickup unit (such as the blur correction lens 12a, the focus lens 12 b, and the image pickup device 14) based on asignal of the adjusting unit 17.

The image information calculating unit 16 includes the image informationdistribution detecting unit 16 a that selects the image informationextraction range (such as the feature point 33 a) where the imageinformation is to be extracted in one frame and that detects thedistribution of the selected image information extraction range.Furthermore, the image information calculation unit 16 includes thefeature point coordinate calculating unit 16 b that obtains thecoordinate of a first image information extraction range (first featurepoint) at the readout time of a first image signal from the imagesignals. In addition, the image information calculation unit 16 includesthe feature point coordinate estimating unit 16 c that estimates thecoordinate of a second image information extraction range (secondfeature point) different from the first feature point based on thecoordinate of the first image information extraction range calculated bythe feature point coordinate calculating unit 16 b. Furthermore, theimage information calculating unit 16 includes the image informationcomparing unit 16 d that compares the coordinate of the second imageinformation extraction range estimated by the feature point coordinateestimating unit 16 c with the coordinate of the second image informationextraction range calculated by the feature point coordinate calculatingunit 16 b. The coordinate of the second image information extractionrange calculated by the feature point coordinate calculating unit 16 bis calculated at the readout time of a second image signal after thepassage of a predetermined time period from the readout time of thefirst image signal.

The signal readout unit 15 reads a region including the imageinformation extraction range (such as the feature point 33 a and therange-finding frame 93 a 1 capturing an object to be focused) withpriority. For details, the signal readout unit 15 performs thedecimating readout for regions other than the region including the imageinformation extraction range, or changes the readout order based on thedistribution of the image information extraction range. The imageinformation distribution detecting unit 16 a averages the coordinates ofthe same image information extraction ranges (i.e., corresponding imageinformation extraction ranges) read in a plurality of frames (forexample, the frames 51, 52, and 53) to detect the distribution of theimage information extraction ranges. Furthermore, the image informationdistribution detecting unit 16 a detects a distribution of each imageinformation extraction range detected in one frame based on a change ofthe coordinates of the same image information extraction ranges (i.e.,corresponding image information extraction ranges) read in the pluralityof frames.

The signal readout unit 15 reads the region including the imageinformation extraction range at approximately constant intervals in oneframe based on the distribution of the image information extractionrange. Furthermore, the signal readout unit 15 reads a region (each ofregions 65 a to 65 e) which includes the image information extractionrange and a region (each of regions 66 a to 66 e) which does not includethe image information extraction range approximately alternately. Inaddition, the signal readout unit 15 divides each of the region whichincludes the image information extraction range and the region whichdoes not include the image information extraction range into regionshaving approximately the same number, and it reads each of the dividedregions approximately alternately.

The image information calculating unit 16 includes the image informationcomplementing unit 16 e that complements image information of an imageinformation lacked portion based on the obtained image information (suchas the image blur correction plots 21 a′ to 24 e′ and the focuscorrection plots 91 a′ to 91 e′). By using the image information, theimage pickup unit can be controlled by a plurality of image signals inone frame, and accordingly it is possible to perform the correction withhigh response. In addition, the image information is obtained by usingthe different image signals in one frame, and accordingly a noise thatis superimposed on an image can be effectively reduced compared with aconventional technology which repeatedly reads the same image signal ina non-destructive method.

Second Embodiment

Next, a second embodiment of this embodiment will be described. Whilethe plurality of feature points in a frame are compared to calculate theimage blur correction plots in the first embodiment, the same featurepoints (i.e., corresponding feature points) between a plurality offrames are compared to obtain the image blur correction plots in thisembodiment. By repeating this operation for a plurality of differentimage signals in one frame, the plurality of image blur correction plotsare calculated in a time during which the frame is captured, and as aresult a correction with high response is achieved.

FIG. 11 is an explanatory diagram of the readout of image signals inthis embodiment. In FIG. 11, image blur correction plots 1101 a′ to 1103e′ are obtained from each of frames 1101, 1102, 1103, and 1104 which isdivided into a plurality of regions “a” to “e” in a direction of animaging-plane height based on an instruction of the signal readout unit15. In FIG. 11, a horizontal axis indicates a time, and an accumulationtime for one frame is a time period 1106 and for example it is 1/30. InFIG. 11, a vertical axis indicates a position in the height direction(imaging-plane height) of an image obtained in each frame, and FIG. 11is a diagram of a rolling readout in an image pickup element with aso-called CMOS structure. Furthermore, the vertical axis indicates anamount of correcting a blur occurring in an image plane.

Each of the image signals in the regions 1101 a to 1101 e is read afterthe passage of the accumulation time period. Image signals 1102 a to1102 e in the subsequent frame is read with a delay corresponding to oneframe (for example 1/30). Then, a coordinate of an image informationextraction range (feature point in the region 1101 a selected by theimage information distribution detecting unit 16 a) obtained from imagesignals in the region 1101 a is compared with a coordinate of the samefeature point obtained from image signals in the region 1102 a. As aresult, an image shift amount (blur amount) on the imaging planeoccurring in one frame can be obtained. The image blur correction plot1101 a′ is obtained based on the image shift amount. Similarly, bycontinuing the comparison of the same feature points between a pluralityof frames such as between the regions 1101 b and 1102 b, and between theregions 1101 c and 1102 c, the image blur correction plots 1101 b′ to1103 e′ are obtained. An image blur correction waveform 1105 is obtainedbased on the image blur correction plots 1101 a′ to 1103 e′ obtained asdescribed above. Since the image blur correction plots obtained by thecomparison of the same feature points between the frames have widesampling intervals each other, it is difficult to perform the image blurcorrection with high accuracy. Accordingly, in this embodiment, acorrelation of different feature points in a frame is also used toimprove the accuracy of the image blur correction.

FIGS. 12A to 12N are explanatory diagrams of the relationship of theimage information extraction ranges in this embodiment, and theyillustrate graphs of improving the accuracy of the image blurcorrection. In each of FIGS. 12A to 12N, a horizontal axis indicates atime, and a vertical axis indicates an image blur correction amount.FIG. 12A illustrates a comparison waveform 1108 a obtained from imagesignals in regions 1101 a, 1102 a, 1103 a, and 1104 a. FIG. 12Billustrates a comparison waveform 1108 b obtained from image signals inregions 1101 b, 1102 b, 1103 b, and 1104 b. FIG. 12C illustrates acomparison waveform 1108 c obtained from image signals in regions 1101c, 1102 c, 1103 c, and 1104 c. FIG. 12D illustrates a comparisonwaveform 1108 d obtained from image signals in regions 1101 d, 1102 d,1103 d, and 1104 d. FIG. 12E illustrates a comparison waveform 1108 eobtained from image signals in regions 1101 e, 1102 e, 1103 e, and 1104e.

The comparison waveforms 1108 a to 1108 e have initial values as imageblur correction plots 1101 a′, 1101 b′, 1101 c′, 1101 d′, and 1101 e′,respectively. Subsequently, the comparison waveform 1108 b overlaps withthe comparison waveform 1108 a. In this case, the comparison waveform1108 b overlaps with reference to the comparison waveform 1108 a suchthat a correlation of both the waveforms is highest. For example, amethod of overlapping the waveforms such that an absolute value of anarea surrounded by both the waveforms is minimized is adopted.Subsequently, with reference to the comparison waveform 1108 boverlapped with the comparison waveform 1108 a, the comparison waveform1108 c overlaps with the waveform. By repeating the processing, asillustrated in FIG. 12F, image blur correction plots 1101 a′ to 1103 e′are arranged. By connecting the image blur correction plots 1101 a′ to1103 e′, as illustrated in FIG. 12G, an image blur correction waveform1107 can be obtained.

As above, since each of the comparison waveforms are obtained and thenthe correlation of them is obtained, the time delay increases by thetime required for obtaining the waveforms and the time required forcalculating the correlation if the image blur correction is performed byusing the correlation. In order to perform the image blur correctionwith small time delay in reality, the image blur correction plot isobtained for each feature point by using a relative coordinate ofdifferent feature points in one frame obtained in FIGS. 12A to 12G.Specifically, a change of the same feature point in the next frame withrespect to a position of the image blur correction plot 1103 a′ in FIG.12G is obtained, and similarly a change of the same feature point in thenext frame with respect to a position of the image blur correction plot1103 b is obtained. FIGS. 12H to 12J illustrate the processing indetail. After each of the comparison waveforms 1108 a to 1108 e overlapsat an appropriate position in FIG. 12F, each of the comparison waveformsare separated again as illustrated in FIGS. 12H and 12I while theposition is maintained. Then, the same feature points between frames arecompared for each feature point. In other words, in FIG. 12H, imagesignals of the region 1104 a in the frame 1104 are compared with imagesignals obtained in the next frame to obtain its change amount, andfurthermore the change amount is added with reference to the image blurcorrection plot 1103 a′ to obtain the image blur correction plot 1104a′. Each of the comparison waveforms obtained by repeating the similaroperations until FIG. 12L is synthesized again to obtain FIG. 12M. Then,based on the newly obtained image blur correction plots 1104 a′ to 1104e′, an image blur correction waveform 1107 is obtained as illustrated inFIG. 12N.

As described above, if the relative coordinate of each feature point ina frame can be obtained once, the image blur correction plots betweenframes may overlap while its relationship is maintained after that, andthe correlation of the comparison waveforms is not necessary.Accordingly, it is possible to perform a highly-accurate image blurcorrection using many image blur correction plots without a time delay.This embodiment is not limited to the method of obtaining therelationship of the different feature points in one frame describedabove based on the correlation of the comparison waveforms obtainedbetween frames, and as described referring to FIGS. 4A, 4B, 5A, and 5B,the method of obtaining the distribution of the feature points inadvance is also applicable. Since the relationship of the respectivefeature points in one frame is known in FIGS. 4A, 4B, 5A, and 5B, eachfeature point in a first frame may be set as an initial value and achange of the feature point between frames may be added to the initialvalue.

FIG. 13 is a flowchart of creating the image blur correction plot inthis embodiment, and it illustrates a subroutine in creating the imageblur correction plot at step S705 of FIG. 7A. First, at step S1301, theimage information calculating unit 16 determines whether a relationshipof the image blur correction plots in one frame described referring toFIGS. 12A to 12G has been already obtained. If the relationship of theimage blur correction plots has been already obtained, the flow proceedsto step S1307. On the other hand, if the relationship of the image blurcorrection plots has not yet obtained, the flow proceeds to step S1302.

At step S1302, the image information comparing unit 16 d obtains achange of the same feature points (image blur correction plots) in apreviously-captured frame and a currently-captured frame. In this case,changes of a plurality of feature points of different regions in oneframe with respect to the same corresponding feature points in theprevious frame are obtained, and as a result, a plurality of image blurcorrection plots are obtained in one frame. Subsequently, at step S1303,the image information calculating unit 16 determines whether theoperation of step S1302 is performed for four frames. If the operationis performed for the four frames, the flow proceeds to step S1304. Onthe other hand, if the operation is not yet performed for the fourframes, the flow returns to step S1302 and the same operation isrepeated. For example, the calculation of the image blur correction plotis repeated by using the frames 1101 to 1104 in FIG. 11.

At step S1304, the image information calculating unit 16 obtainscomparison waveforms of FIGS. 12A to 12E based on the image blurcorrection plots obtained at steps S1302 and S1303. Subsequently, atstep S1305, the image information distribution detecting unit 16 aoverlaps the comparison waveforms obtained at step S1304 each other toincrease the correlation (FIGS. 12F and 12G). Subsequently, at stepS1306, the image information distribution detecting unit 16 a detectsthe relationship of the image blur correction plots obtained as a resultof overlapping the waveforms at step S1305. Accordingly, therelationship of the image blur correction plots obtained from thedifferent feature points in one frame can be recognized. Subsequently,at step S1307, the feature point coordinate calculating unit 16 b adds achange amount of the feature point between the frames obtained by thecomparison with the subsequent frame as described referring to FIGS. 12Hto 12M to an initial value that corresponds to the relationshipdescribed above to obtain a subsequent image blur correction plot. Whenthe subsequent image blur correction plot is obtained at step S1307,this subroutine is finished and the flow proceeds to step S706.

As described above, the feature point coordinate calculating unit 16 bcompares the plurality of image information extraction ranges betweenframes for each of the same image information extraction range (i.e.,corresponding image information extraction ranges such as rangesincluded in the regions 1101 a, 1102 a, 1103 a, and 1103 a). Then, thefeature point coordinate calculating unit 16 b obtains the plurality ofcomparison waveforms (such as waveforms 1108 a to 1108 d) based on aresult of the comparison, and it obtains a target adjustment value basedon the relationship of the plurality of comparison waveforms.Specifically, the image information distribution detecting unit 16 aobtains the relationship between the different image informationextraction ranges (such as ranges included in the regions 1101 a to 1101e) in one frame based on the plurality of comparison waveforms (such aswaves 1108 a to 1108 d). The feature point coordinate calculating unit16 b obtains a target adjustment value in one frame for the subsequentframe based on the relationship between the different image informationextraction ranges in one frame obtained by the image informationdistribution detecting unit 16 a. As a result, the image pickup unit canbe controlled by a plurality of image signals in one frame, andaccordingly a correction with high response can be performed.

Third Embodiment

Next, a third embodiment of the present invention will be described. Inthis embodiment, a case in which photographing times (signalaccumulation times) of a plurality of frames are different from eachother will be described. The case in which the photographing times ofthe plurality of frames are different from each other means that forexample a case in which a state (live view) in which an object iscaptured at the step of image capturing preparation is to proceed to astill image capturing or a case in which a brightness of the object isgreatly changed during the live view or a moving image capturing. Inthis embodiment, especially a case of proceeding to the still imagecapturing will be described. In the moving image capturing or the liveview, the image is captured by a short photographing time to some extentsince the motion of the object is prioritized. On the other hand, in thestill image capturing, it is preferred that more appropriate exposurecontrol is performed to obtain an image with less noise. Accordingly, inmany cases, an exposure time of the still image may be increasedcompared with a moving image (the photographing time of the still imagemay be decreased compared with the moving image in order to stop themotion of the object in some cases).

FIGS. 14A and 14B are explanatory diagrams of the operation of thesignal readout unit which can be adapted to the still image capturing inthis embodiment. Referring to FIG. 14A, a method of uniformly obtainingimage information in one frame will be described by using an example ofthe image blur correction in the image capturing condition describedabove. Image blur correction plots 1401 a′ to 1401 e′ are obtained fromregions 1401 a to 1401 e depending on the readout in a frame 1401. Atthis time, a signal accumulation time 1406 in one frame is for example1/30 sec. In FIG. 14A, a signal readout time 1411 in this case is set toapproximately the same as the signal accumulation time 1406, but thisembodiment is not limited thereto. For example, a signal readout time1411 may be set to be shorter than the signal accumulation time 1406.

In a frame 1402 in which the speed of the signal readout is fast, thesignal accumulation time in the frame 1402 is set in order to performthe still image capturing in a next frame 1403. When a signalaccumulation time 1407 which is suitable for the still image capturingis for example 1/10 sec, a signal readout time 1412 which isapproximately the same as the time is set in the previous frame 1402. Inother words, in the frame 1402, the setting time of the accumulationstart scan is set according to a time 1411 of readout 1408 (for example,1/30 sec) in the previous frame 1401, and a time 1412 of a readout 1409is set to 1/10 sec. A region in the frame 1401 are divided(segmentalized) into regions in the frame 1402. This is because the time1412 of the readout 1409 is longer than the time 1411 of the readout1408 and an interval in which image signals are obtained by the samenumber of regions increases, and thus the complementation is needed. Inother words, the image information distribution detecting unit 16 adivides (segmentalizes) the readout region of an image prior to thestill image capturing.

In FIG. 14A, regions of the frame 1402 are obtained by dividing eachregion of the frame 1401 into two equal parts to double the number ofregions. Image blur correction plots 1402 a 1′, 1402 a 2′ to 1402 e 1,and 1402 e 2′ are respectively obtained based on regions 1402 a 1, 1402a 2 to 1402 e 1, and 1402 e 2 in the frame 1402. In the frame 1402,while the accumulation times of the regions 1402 a 1, 1402 a 2 to 1402 e1, and 1402 e 2 are different from each other, it is not a problem sincethey are used to obtain the image blur correction plots based on theimage signals instead of obtaining a viewing image.

Furthermore, the image information complementing unit 16 e performs thegain-up and the gain-down processing of image signals depending adifference of the accumulation times in a frame in order to stabilizethe image signals. As described above, in the frame 1402, the number ofthe regions to read image signals increases compared with the frame1401, and accordingly it is necessary to add feature points. The addedfeature points are, as described referring to FIG. 7B, obtained by theimage information distribution detecting unit 16 a that adds remainingfeature points previously detected as image information extractionranges. As described above, the image blur correction plots are obtainedto perform the image blur correction based on the image signals read inthe frames 1401 and 1402. The way to obtain the image blur correctionplots is the same as that in the first embodiment, and accordinglydescriptions thereof are omitted. Also with respect to a frame 1403 inwhich a still image is to be captured, by using the same number ofregions in the frame 1402, widening intervals for obtaining imagesignals is prevented.

The setting time of an accumulation start operation in the frame 1403 isset according to a time 1412 (for example, 1/10 sec) of readout 1409 inthe previous frame 1402, and a time 1413 of readout 1410 is also set tothe same speed ( 1/10 sec). Accordingly, the signal accumulation time1407 in each region is adjusted (aligned). Then, image blur correctionplots 1403 a 1′, 1403 a 2′ to 1403 e 1′, and 1403 e 2′ are obtainedbased on respective regions 1403 a 1, 1403 a 2 to 1403 e 1, and 1403 e 2during the time 1413 along the signal readout 1410 for completion ofsignal accumulation in each region. Thus, the image blur correction isperformed during the still image capturing.

As described above, each of the times 1412 and 1413 at which signals areread in the frame 1403 to perform the still image capturing and in theprevious frame 1402 is set to have a length approximately the same asthat of the signal accumulation time. As a result, a signal readout 1409in the frame 1402 and a signal readout 1410 in the frame 1403continuously start with the signal readout 1408 in the previous frame1401, and accordingly the image signals can be obtained continuouslyduring the still image capturing. In other words, the signal readoutunit 15 changes a signal readout time according to the signalaccumulation time required for the still image capturing (i.e.,depending on the signal accumulation time). When the readout of theframe 1403 which is used to obtain the still image is completed, for thenext frame 1404, the number of divided regions of the image and thereadout time are restored to the same number of divided regions and thesame readout time as those in the frame 1401. Then, image blurcorrection plots 1404 a′ and 1404 e′ are obtained from regions 1404 a to1404 e. Furthermore, in a next frame 1405, image blur correction plots1405 a′ and 1405 e′ are obtained from regions 1405 a to 1405 e.According to the operations described above, an image blur correctionwaveform 1414 can be obtained continuously from the frame 1401 prior tothe still image capturing to the frame 1405 subsequent to the stillimage capturing, and thus it is possible to perform the image blurcorrection during the still image capturing. As above, the case in whichthe feature points are uniquely distributed in an image (screen) isdescribed.

Next, as illustrated in FIG. 6A, a case in which the feature points areconcentrated on the center of the image will be described. For example,it is assumed that the feature points are distributed in regions 1401 b,1401 c, and 1401 d in FIG. 14A. In this case, in order to obtain imageblur correction plots at equal intervals, as described referring to FIG.6C, it is necessary to change an order of the readout of regions.Referring to FIG. 14B, a change of the readout order in a case in whicha situation proceeds from the live view to the still image capturingwill be described. Each of frames, regions, and image blur correctionplots in FIG. 14B are the same as those in FIG. 14A, and accordingly thesame signs are indicated.

First, with respect to the frame 1401, the image informationdistribution detecting unit 16 a sets the readout order of image signalsof the image pickup device 14 by the signal readout unit 15 based on adetection result of the feature point distribution. If the featurepoints are concentrated in the regions 1401 b to 1401 d, the readouts ofthe regions are performed at approximately equal intervals during atotal readout time. In other words, regions in which feature points aredistributed and regions in which any feature points are distributed areread alternately in order from the region 1401 c (with a feature point),the region 1401 a (without any feature point), the region 1401 b (with afeature point), the region 1401 e (without any feature point), and theregion 1401 d (with a feature point). As described above, with respectto the frame 1401, image blur correction plots 1401 c′ and 1401 b′ areobtained based on the regions 1401 b to 1401 d depending on the readout.Black plots in FIG. 14B are, as described referring to FIG. 2, imageblur correction plots which are predicted by the image informationcomplementing unit 16 e by using the adaptive filter or the linearprediction. For the same reason as that of FIG. 14A, the regions in theframe 1402 are obtained by dividing the region in the frame 1401. Inaddition, image signals are read from regions 1402 c 1, 1402 b 2, 1402 b1, 1402 c 2, and 1402 d 1 to obtain image blur correction plots 1402 c1′, 1402 b 2′, 1402 b 1′, 1402 c 2′, and 1402 d 1′.

Also with respect to the frame 1403 in which the still image is to becaptured, by using the same number of regions in the frame 1402,widening intervals for obtaining image signals is prevented. Imagesignals are read from the regions 1403 c 1, 1403 b 2, 1403 b 1, 1403 c2, and 1403 d 1 to obtain the image blur correction plots 1403 c 1′,1403 b 2′, 1403 b 1′, 1403 c 2′, and 1403 d 1′. Thus, the image blurcorrection is performed during the still image capturing. When thereadout of the frame 1403 for obtaining the still image is completed,also with respect to a next frame 1404, image signals are read fromregions 1404 c 2 and 1404 b 2 by the same number of divided regions asthat in the frame 1403 to obtain image blur correction plots 1404 c 2′and 1404 b 2′.

FIGS. 15A and 15B are explanatory diagrams of the operation of anothersignal readout unit which can be adapted to the still image capturing inthis embodiment. Also with respect to the frame 1404, the readout regionis divided since the frame 1404 is read continuously from the readout ofeach region in the frame 1403. However, as illustrated in FIG. 15A, bysetting the frames 1403 and 1404 in a discontinuous manner once, thesegmentalization of the divided regions can be avoided. In FIG. 15A, thereadout between the frames 1401 and 1402 b is also discontinuous, andthus an accumulation time of each of regions 1402 a 1 to 1402 e 2 can beadjusted (aligned). Since a blur amount during the accumulation time canbe aligned by the adjustment of the accumulation time, image blurcorrection plots can be obtained with higher accuracy.

A method of accumulation like the frame 1402 in FIG. 15A can also beapplied to FIG. 14A. Like the frame 1402 in FIG. 14A, even when theaccumulation time increases in the readout order of image signals in theregions 1402 a 1 to 1402 e 2, the accumulation times can be aligned byadopting the configuration of the frame 1402 in FIG. 15B. As describedabove, the frame 1402 prior to the frame 1403 as a still image capturingframe is read discontinuously with the previous frame 1401, andaccordingly the accuracy of the image blur correction can be improved.Furthermore, a frame 1404 subsequent to the frame 1403 as the stillimage capturing frame is read discontinuously with the previous frame1403, and accordingly the segmentalization of regions can be avoided andalso the readout load can be reduced.

Returning to FIG. 14B, in a frame 1405, image signals are read fromregions 1405 c, 1405 b, and 105 d to obtain image blur correction plots1405 c′, 1405 b′, and 1405 d′, respectively. According to the operation,even if the feature points are concentrated on the center of the imageor the like, an image blur correction waveform 1414 can be obtainedcontinuously from the frame 1401 prior to the still image capturing tothe frame 1405 subsequent to the still image capturing, and accordinglyit is possible to perform the image blur correction during the stillimage capturing. As described above, while the way of adaption of thestill image capturing during the still image capturing is describedreferring to FIGS. 14A to 15B, the focusing can also be performed by thesimilar processing.

FIGS. 16A and 16B are explanatory diagrams of the operation of anothersignal readout unit which can be adapted to the still image capturing inthis embodiment. FIG. 16A is an explanatory diagram of the focusing, andas illustrated in FIG. 9A, processing on condition that therange-finding frames (range-finding frames in which an object to befocused exists) selected by the image information distribution detectingunit 16 a are distributed on an entire image (screen) is illustrated.

The frames 1401, 1402, 1404, and 1405 are frames for the live view, andthe frame 1403 is a frame for the still image capturing. Image signalsare obtained from each region in each frame, and focus correction plots1401 a′ to 1405 e′ indicating focus states in the respective regions areobtained based on information (for example, phase differenceinformation) of the range-finding frames included in the region. Theconcept of the segmentalization of the region prior to the still imageframe to complement the readout interval is similar to the image blurcorrection in FIGS. 14A to 15B. The signal readout unit 15 changes thesignal readout time (readout speed) according to the signal accumulationtime required for the still image capturing.

FIG. 16A is different from each of FIGS. 14A to 15B in an accumulationstart scan 1601 (signal reset) in the frame 1402 and a speed of areadout 1602 in the frame 1404. However, this embodiment is not limitedto the accumulation start scan 1601 and the readout 1602, and theaccumulation start, and alternatively, the readout as illustrated inFIG. 14A or 15B may be adopted. As illustrated in FIG. 9C, if therange-finding frames selected by the image information distributiondetection unit 16 a are concentrated on the center of the image, thereadout order is changed as illustrated in FIG. 16B and focus statedetection signals are set to be obtained at approximately the sameintervals. In other words, regions in which selected range-findingframes are distributed and regions in which the selected range-findingframes are not distributed are read alternately in order from theregions 1401 c (with a selected range-finding frame), the region 1401 a(without any selected range-finding frame), the region 1401 b (with aselected range-finding frame), the region 1401 e (without any selectedrange-finding frame), and the region 1401 d (with a selectedrange-finding frame).

As described above, the focus correction plots 1401 c′ and 1401 b′indicating the focus states are obtained from the regions 1401 b to 1401d depending on the readout in the frame 1401. Black plots in FIG. 16Bare, as described referring to FIG. 2, image blur correction plots whichare predicted by the image information complementing unit 16 e using theadaptive filter or the linear prediction. As an example illustrated inFIG. 16B, the readout order of the frame 1402 is changed from that inFIG. 16A, but this embodiment is not limited thereto and for example itmay be configured as illustrated in FIG. 15A.

This embodiment can be adapted to still image capturing in addition toeach of the first and second embodiments. This flow is performed at thestep of setting the readout order at step S703 in FIG. 7A and at stepS1002 in FIG. 10A, and its detail is illustrated in FIG. 17. FIG. 17 isa flowchart of setting the readout order in this embodiment. FIG. 17illustrates a subroutine in setting the readout order at steps S703 andS1002.

First, at step S1701, the CPU 111 determines whether a still image is tobe captured. For example, this can be determined by detecting a signal,input to the CPU 111, from the operating unit 112 in FIG. 1. If thestill image is to be captured, the flow proceeds to step S1702. On theother hand, if the still image is not to be captured, the flow proceedsto step S1704. At step S1702, the image information calculating unit 16reads an exposure time (signal accumulation time) required for capturingthe still image from the CPU 111. Then, the image informationdistribution detecting unit 16 a controls the signal readout unit 15 soas to perform the signal readout depending on the exposure time. Forexample, if the exposure time is 1/10 sec, the readout time is also setto 1/10 sec. If the exposure time is ½ sec, the readout time is also setto ½ sec. Subsequently, at step S1703, the image informationdistribution detecting unit 16 a segmentalizes the readout regions suchthat the image information can be obtained at desired intervals such asfor each 2 ms during the readout time set at step S1702.

At step S1704, the image information calculating unit 16 determineswhether there is a flag indicating a change of the readout order at stepS7014 in the flowchart of FIG. 7B. If this flag exists, the flowproceeds to step S1705. On the other hand, if the flag does not exist,this subroutine is finished and the flow proceeds to step S704 in FIG.7A or step S1003 in FIG. 10A. While the distribution of the featurepoints are described in FIG. 7B, the image information distributiondetecting unit 16 a may set the flag based on a distribution state ofrange-finding frames which coincide with an object to be focused.Accordingly, while “IS THERE FLAG AT FEATURE POINT?” is described asstep S7031 in FIG. 7F, the description is changed to “IS THERE FLAG ATSELECTED POINT?” (with respect to the image information extractionrange) at step S1704 in FIG. 17 considering both the image blurcorrection and the focusing. Steps S1705 and S1706 in FIG. 17 are thesame as steps S7032 and S7033 in FIG. 7F, respectively, and accordinglydescriptions thereof are omitted. When sorting of the readout order iscompleted at step S1706, this subroutine is finished and the flowproceeds to step S704 in FIG. 7A or step S1003 in FIG. 10A.

As described above, this embodiment can be adapted to the situation inwhich a photographing time is changed from a photographing preparationstate to a still image capturing, and the signal readout unit 15segmentalizes the readout region of an image prior to the still imagecapturing. The image information complementing unit 16 e performs a gainadjustment of an image signal depending on an accumulation timedifference in one frame. The signal readout unit 15 changes a signalreadout time in accordance with the signal accumulation time requiredfor the still image capturing. By performing the processing, a stableimage signal can be obtained to drive the image pickup unit with highaccuracy even if the photographing time such as the still imagecapturing changes.

Fourth Embodiment

Next, a fourth embodiment of this embodiment will be described. Areadout time 1413 is required in addition to a signal accumulation time1407 of FIG. 14A as a time until the still image capturing is completedin the third embodiment. This is because the speed of readout 1410 isadjusted to the signal accumulation time 1407 and the image signals areread over an entire accumulation time. This embodiment is directed to amethod of reducing the photographing time.

FIGS. 18A and 18B are explanatory diagrams of the operation of thesignal readout unit which can be adapted to the still image capturing inthis embodiment. In FIG. 18A, in a frame 1801, similarly to the frame1401 illustrated in FIG. 14A, image blur correction plots 1801 a′ to1801 e′ are acquired depending on the readout of image signals fromregions 1801 a to 1801 e, respectively. In this embodiment, the imageprocessing unit 19 synthesizes an image of a frame 1802 with an image ofa next frame 1803 to acquire a still image. In the frame 1802, similarlyto the frame 1402 illustrated in FIG. 14A, a signal accumulation timefor the frame 1802 is set to perform the still image capturing. Alsowith respect to the frame 1802, similarly to the frame 1402, the regionis segmentalized (divided) for the purpose of shortening intervals toobtain image signals. Image blur correction plots 1802 a 1′, 1802 a 2′to 1802 e 1′, and 1802 e 2′ are acquired from regions 1802 a 1, 1802 a 2to 1802 e 1, and 1802 e 2 in the frame 1802.

As can be seen in FIG. 18A, with respect to the frame 1802, anaccumulation time 1806 varies depending on the imaging-plane height, andthe accumulation time decreases with increasing the imaging-planeheight. In other words, unevenness of exposure occurs in the obtainedimage. Also with respect to a frame 1803, by using the same number ofregions in the frame 1802, widening intervals for obtaining imagesignals is prevented. A setting time of an accumulation start scan inthe frame 1803 is aligned to the readout time in the previous frame 1802to capture images continuously. However, the readout time of the frame1803 is set to be equal to the setting time of the accumulation startscan in the frame 1802 (equal to the readout time in the frame 1801 inFIG. 18A). Accordingly, as can be seen in FIG. 18A, with respect to theframe 1803, a signal accumulation time 1807 varies depending on theimaging-plane height, and the accumulation time increases withincreasing the imaging-plane height. In other words, the unevenness ofexposure occurs in the obtained image.

Then, image blur correction plots 1803 b 2′, 1803 c 2′, 1803 d 2′, and1803 e 2′ are acquired from image signals in regions 1803 b 2, 1803 c 2,1803 d 2, and 1803 e 2, respectively. With respect to the frame 1803,only the image signals in the regions 1803 b 2, 1803 c 2, 1803 d 2, and1803 e 2 are used. Accordingly, the region is not segmentalizeddifferently from the frame 1802, it may be divided into regions havingapproximately the same number of regions in the frame 1801.Alternatively, the image blur correction plots may be obtained morefinely by using image signals in regions 1803 a 1, 1803 a 2, 1803 b 1,1803 c 1, 1803 d 1, and 1803 e 1.

With respect to a frame 1804, image signals in regions 1804 a to 1804 eare read out to obtain image blur correction plots 1804 a′ to 1804 e′,respectively. Then, based on the image blur correction plots obtained inthe frames 1801 to 1804, the image blur correction lens 12 a is drivento perform the image blur correction. The image processing unit 19synthesizes images obtained from the frames 1802 and 1803 to obtain astill image. As described above, although the unevenness of exposureoccurs in both the frames 1802 and 1803, an image without the unevennessof exposure can be obtained by mutual complementation since theunevenness directions are opposite to each other. When synthesizing theframes 1802 and 1803, the image processing unit 19 aligns positions ofthe images by using a known alignment technology, and accordingly animage shift between the two frames can be canceled (or reduced).

As described above, the image processing unit 19 synthesizes the twoframes to obtain a still image, and accordingly the photographing timecan be reduced. The signal readout unit 15 adjusts the adjacent readouttime between the frames 1802 and 1803 to the signal accumulation time(i.e., increase the readout time with increasing the accumulation time),and accordingly it can obtain the image blur correction plots uniformlyduring the still image capturing as well. While FIG. 18A describes theimage blur correction as an example, a stable focusing can be performedby similar processing in focus state detection.

While the case in which the feature points are uniformly distributed inthe image, subsequently, a case in which range-finding frames used forthe focusing are concentrated at the center of the image as illustratedin FIG. 9C will be described. In order to obtain plots indicating thefocus states with equal intervals in this case, as described referringto FIG. 14B, the readout order of regions is changed.

Referring to FIG. 18B, the focusing during the still image capturing inthis embodiment will be described. In FIG. 18B, regions in whichselected range-finding frames are distributed and regions in which theselected range-finding frames are not distributed are read alternatelyin order from a region 1802 c 1 (with a selected range-finding frame), aregion 1802 a 1 (without any selected range-finding frame), a region1802 b 2 (with a selected range-finding frame), and a region 1802 a 2(without any selected range-finding frame). Subsequently, a region 1802b 1 (with a selected range-finding frame), a region 1802 d 2 (withoutany selected range-finding frame), a region 1802 c 2 (with a selectedrange-finding frame), a region 1802 e 1 (without any selectedrange-finding frame), a region 1802 d 1 (with a selected range-findingframe), and a region 1802 e 2 (without any selected range-finding frame)are read out in this order. As described above, plots 1802 c 1′, 1802 b2′, 1802 b 1′, 1802 c 2′, 1802 d 1′, 1802 a 2′, and 1802 c 2′ indicatingthe focus states are obtained depending the readout in the frame 1802.Black plots in FIG. 18B are, as described referring to FIG. 2, focuscorrection plots that are predicted by the image informationcomplementing unit 16 e using the adaptive filter or the linearprediction. In this embodiment, by using the method of synthesizing twoframe images to obtain a still image, the readout time can be reduced.While the focus state detection is described in FIG. 18B as an example,this embodiment can be applied also to the image blur correction.

FIGS. 19A and 19B are flowcharts of the image blur correction adapted tothe still image capturing in this embodiment. FIG. 19A is a flowchart ofadding steps (S1901 to S1910) especially for the still image capturingto the flow (steps S701 to S711) of the image blur correction in FIG.7A, and the same steps as those in FIG. 7A are indicated by the samenumber and descriptions thereof are omitted. For simple explanations,elements which are not directly relevant to this embodiment are omitted.

At step S1901, until a user presses a release button provided in theimage pickup apparatus 100 (camera) for example to instruct a start ofthe still image capturing, the flow returns to step S701 or step S704through step S711 to be repeated. If the operation of instructing thestill image capturing is performed at step S1901, the flow proceeds tostep S1902. At step S1902, similarly to step S704, the signal readoutunit 15 reads the image signals for each region in the readout order setat step S703. Subsequently, at step S1903, similarly to step S705, theimage information comparing unit 16 d compares the read image signalsfor each region. Then, the image information calculating unit 16 createsimage blur correction plots.

Subsequently, at step S1904, similarly to step S706, the adjusting unit17 creates a target adjustment value (target image blur correctionvalue) for the image blur correction based on the obtained image blurcorrection plots. Subsequently, at step S1905, similarly to step S707,the drive unit 18 a drives the image blur correction lens 12 a based onthe target adjustment value obtained by the adjusting unit 17.Subsequently, at step S1906, similarly to step S708, the signal readoutunit 15 reads image signals in the next region. Subsequently, at stepS1907, similarly to step S709, the signal readout unit 15 determineswhether image signals in regions required for the image formation in oneframe is read. If the image signals are read out, the flow proceeds tostep S1908. On the other hand, if any region to be read in one frameremains, the flow returns to step S1902, and image signals arerepeatedly read out and the image blur correction is performed based onthe image signals.

At step S1908, the image processing unit 19 or the image informationcalculating unit 16 determines whether the processing of prescribedframes is completed. The prescribed frames are two frames of the frames1802 and 1803 illustrated in FIG. 18A, and they may three or more framesif the captured still image is constituted by using more frames.Subsequently, at step S1909, the image processing unit 19 performsalignment processing such as an alignment of positions of the featurepoints in an image for a displacement between the captured frames toreduce or remove the displacement between the frames. Subsequently, atstep S1910, the image processing unit 19 synthesizes (combines) thealigned frames to obtain a still image.

FIG. 19B is a flowchart of adding steps (S1911 to S1920) especially forthe still image capturing to the flow (steps S1001 to S1010) of thefocus correction in FIG. 10A, and the same steps as those in FIG. 10Aare indicated by the same number and descriptions thereof are omitted.

At step S1911, until the user presses the release button provided in theimage pickup apparatus 100 (camera) for example to instruct a start ofthe still image capturing, the flow returns to step S1001 or step S1003through step S1010 to be repeated. If the operation of instructing thestill image capturing is performed at step S1911, the flow proceeds tostep S1912. At step S1912, similarly to step S1003, the signal readoutunit 15 reads the image signals in each region in the readout order setat step S1002. Subsequently, at step S1913, similarly to step S1004, theimage information calculating unit 16 detects a focus state for eachregion of the read image signals, and it creates focus correction plotsbased on the instruction of the signal readout unit 15.

Subsequently, at step S1914, similarly to step S1005, the adjusting unit17 creates a target adjustment value (target focus correction value) forthe image blur correction based on the obtained image blur correctionplots. Subsequently, at step S1915, similarly to step S1006, the driveunit 18 a drives the focus lens 12 b based on the target adjustmentvalue obtained by the adjusting unit 17. Subsequently, at step S1916,similarly to step S1007, the signal readout unit 15 read image signalsin the next region. Subsequently, at step S1917, similarly to stepS1008, the signal readout unit 15 determines whether the image signalsin regions required for the image formation in one frame are read out.If the image signals are read out, the flow proceeds to step S1918. Onthe other hand, if any region to be read in one frame remains, the flowreturns to step S1912, and image signals are repeatedly read out and thefocus correction is performed based on the image signals.

At step S1918, the image processing unit 19 or the image informationcalculating unit 16 determines whether the processing of prescribedframes is completed. The prescribed frames are two frames of the frames1802 and 1803 illustrated in FIG. 18B, and they may three or more framesif the captured still image is constituted by using more frames.Subsequently, at step S1919, the image processing unit 19 performsalignment processing such as an alignment of positions of the featurepoints in an image for a displacement between the captured frames toreduce or remove the displacement between the frames. Subsequently, atstep S1920, the image processing unit 19 synthesizes (combines) thealigned frames to obtain a still image.

As illustrated in FIG. 18A, the signal readout unit 15 sets the signalreadout time to be delayed with respect to the setting time of theaccumulation start scan in one frame, and accordingly a totalphotographing time can be reduced with the accumulation time keptconstant. In this processing, although the unevenness of exposure occursdue to the difference of the accumulation time on the top and the bottomof the image, two images which include unevenness of exposure indirections opposite to each other are synthesized, and accordingly itcan be complemented. In other words, the signal readout unit 15generates the unevenness of exposure to perform the image blurcorrection or the focus correction during the still image capturing.

FIGS. 20A and 20B are explanatory diagrams of the operation of anothersignal readout unit which can be adapted to the still image capturing inthis embodiment. If it is assumed that the unevenness of exposure iscomplemented by gain-up of image signals, as illustrated in FIG. 20A, astill image can be created only by the frame 1802, and accordingly theimage blur correction can be performed during the still image capturing.In this case, the occurring unevenness of exposure has a constant amountalong the vertical direction of the image. Accordingly, in order tocomplement it, for example the gain of image signals increases withapproaching the upper side of the image, the gain is not corrected atthe center of the image, and conversely the gain of image signalsdecreases with approaching the lower side of the image, and as a resultthe exposure for the entire image can be stabilized. Alternatively, as amethod of photographing an image in which the unevenness of exposuredoes not occur, as illustrated in FIG. 20B, the number of framesincreases depending on the accumulation time, and the frames aresynthesized to obtain a still image.

In FIG. 20A, after images of frames 2002 to 2005 with gray patterns arecaptured, the images are aligned and synthesized according to stepsS1909 and S1910 in FIG. 19A. Accordingly, a total photographing time2008 does not increase with respect to a signal accumulation time 2007in one frame (i.e., long signal readout time is not necessary). Inaddition, the image blur correction lens 12 a can perform the image blurcorrection based on the image blur correction waveform 2007 obtained byeach of image blur correction plots 2001 a′ to 2006 e′ during the stillimage capturing. As described above, since it is not necessary toincrease the image signal readout time as a result of increasing thenumber of synthesized images, the image blur correction plots can bedensely obtained. Accordingly, the segmentalization (division) of aregion in one frame is not necessary.

As described above, the image processing unit 19 can reduce thephotographing time by synthesizing a frame in which a target adjustmentvalue is obtained based on a plurality of pieces of image informationwith a subsequent frame to obtain a still image. The image informationdistribution detecting unit 16 a can obtain the image blur correctionplots uniformly by setting the adjacent readout time between frames tocorrespond to the accumulation time (increasing the readout time withincreasing the accumulation time). The signal readout unit 15 delays thesignal readout time compared with the setting time of the accumulationstart scan in one frame, and accordingly a total photographing time canbe reduced with the accumulation time kept constant. Furthermore, for aframe next to the frame in which the signal readout time is delayedcompared with the setting time of the accumulation start scan in oneframe, the signal readout unit 15 sets the signal readout time to beearlier than the setting time of the accumulation start scan in oneframe. As a result, the total photographing time can be reduced with theaccumulation time kept constant.

Fifth Embodiment

Next, a fifth embodiment of the present invention will be described.Also in the fifth embodiment, the photographing time increases becausethe time required for the image accumulation start and the signalreadout is added to the actual image accumulation time. As a method ofseparating the accumulation start time and the readout time from theactual signal accumulation time, it is considered that for example themechanical shutter 14 a illustrated in FIG. 1 is used.

FIGS. 21A and 21B are explanatory diagrams of the operation of thesignal readout unit which can be adapted to the still image capturing inthis embodiment. FIG. 21A illustrates an example in which the signalreadout unit 15 electronically resets the accumulation of signals whenstarting the accumulation and then the mechanical shutter 14 a shieldslight so as to prevent the object light beam from reaching the imagepickup device 14 to align the photographing time with the signalaccumulation time. A line segment 2101 indicates a line at which thesignal readout unit 15 resets image signals, and thus image signals 2102in an interval indicated by meshed lines in the frame 1802 are reset.Accordingly, at the time of line segment 2101, the image signals cannotbe read out.

Image information cannot be obtained in a region 1802 a 1 of the frame1802 since the signal accumulation time is short. However, based onimage blur correction plots 1801 a′ to 1801 e′ obtained from a previousframe 1801, the image information complementing unit 16 e can predict animage blur correction plot 1802 a 1′ (indicated by a black circle) readin the region 1802 a 1. A region 1802 a 2 in the frame 1802 has a signalaccumulation time which is longer than that of the region 1802 a 1, butin the region 1802 a 2, a sufficient signal accumulation cannot beperformed. Accordingly, the gain-up of the signals is performed toobtain the image blur correction plot 1802 a 2′ (indicated by a graycircle). In subsequent regions 1802 b 1 to 1802 e 2, the signalaccumulation times are sufficient, and accordingly image blur correctionplots 1802 b 1′ to 1802 e 2′ can be obtained.

In a next frame 1803, the mechanical shutter 14 a is closed at the timeof a line segment 2103 to shield the object light beam entering theimage pickup device 14 until a next line segment 2104. Accordingly, inthe interval indicated by the meshed lines 2105, the image informationfrom the object light beam cannot be obtained. However, in regions 1803a 2, 1803 b 2, and 1803 c 2, sufficient image information can beobtained before the light is shielded, and accordingly image blurcorrection plots 1803 a 2′, 1803 b 2′, and 1803 c 2′ can be obtained. Inother words, the image blur correction can be performed continuously byusing image signals read while the mechanical shutter 14 a shields thelight. However, the gain-up is performed on image signals of a region1803 d 2 in the frame 1803 to obtain image blur correction plot 1802 d2′ (indicated by gray circle) due to the short signal accumulation time.

A region 1803 e 2 in the frame 1803 has a shorter signal accumulationtime, and accordingly the image information cannot be obtained. However,the image information complementing unit 16 e can predict an image blurcorrection plot 1803 e 2′ (indicated by a black circle) read in a region1802 e 2 based on image blur correction plots 1803 a 2′, 1803 b 2′, 1803c 2′, and 1803 d 2′ obtained in advance. Similarly, image blurcorrection plots 1804 a′ and 1804 b′ are obtained by the prediction andthe gain-up, respectively.

FIG. 21B is a timing chart of illustrating a timing of the start ofaccumulation and the signal readout in each frame, the reset of theaccumulation signals, the shield of the light entering the image pickupdevice 14 by the mechanical shutter 14 a. In FIG. 21B, referencenumerals 1801 s, 1802 s, 1803 s, and 1804 s denote time periods ofstarting signal accumulation in frames 1801 to 1804, respectively, andthey are indicated by white circles. In FIG. 21B, reference numerals1801 r, 1802 r, 1803 r, and 1804 r denote time periods of readingsignals in the frames 1801 to 1804, respectively, and they are indicatedby black circles. A line segment 2106 indicates a timing 2108 at whichthe signal readout unit 15 resets image signals prior to the still imagecapturing, and the rest of the signals are performed immediately afterthe time period 1801 r of the signal readout in the frame 1801.Accordingly, in the frame 1802, information obtained by starting thesignal accumulation prior to the reset timing 2108 is also reset. A linesegment 2107 indicates a timing 2109 at which the mechanical shutter 14a shields the light entering the image pickup device 14, and the lightshield is performed immediately after the time period 1803 r of thesignal readout in the frame 1803 starts. Accordingly, the image pickupdevice 14 cannot accumulate image signals during the time period of thelight shield.

As described above, although a part of the image signals cannot beobtained due to the signal reset and the mechanical shutter 14 a, thephotographing time can be set to approximately the same as the signalaccumulation time 1807. While FIG. 21A describes the image blurcorrection as an example, this embodiment can be applied also to thefocusing and a situation in which feature points or objects to befocused are concentrated on the center as illustrated in FIG. 18B. Withrespect to the reset of the signal accumulation by the signal readoutunit 15, the object light beam entering the image pickup device 14 maybe shielded by using the mechanical shutter 14 a during the interval upto the line segment 2101.

As described above, in this embodiment, an accumulation control unit(the signal readout unit 15 and the mechanical shutter 14 a) thatcontrols the start of accumulation of a frame which is first captured ina plurality of frames synthesized by the image processing unit 19 may beprovided. A signal readout control unit (mechanical shutter 14 a) thatcontrols the signal readout of a frame which is captured last in theplurality of frames may be provided. By providing at least one of theaccumulation control unit or the readout control unit, the reduction ofthe photographing time can be achieved. By performing the image blurcorrection using the read image signals while the mechanical shutter 14a as the readout control unit shields the light entering the imagepickup device 14, the image blur correction can be performedcontinuously.

Sixth Embodiment

Next, a sixth embodiment of the present invention will be described. Ineach of the first to fifth embodiments, the drive control of the imageblur correction lens 12 a or the focus lens 12 b is performed by usingthe image signals obtained from each frame. On the other hand, thisembodiment describes an example in which a target adjustment value ofthe image blur correction lens 12 a or the focus lens 12 b is createdbased on the image signals obtained from each frame, and in reality theyare driven after the frame. For example, when the focus lens 12 b iscontinuously driven during capturing a still image, it may be impossibleto perform both the drive and the processing of detecting a focus statebased on the image signals at the same time in view of electricity. Inthis example, a case in which a continuous photography of a still imageis considered.

FIGS. 22A to 22C are explanatory diagrams of a signal readout and a lenscontrol in this embodiment. In FIG. 22A, plots 2201 a′ to 2201 e′ thatindicate focus states in each of regions 2201 a to 2201 e obtained froma frame 2201 are disposed on a desired focus lens driving locusindicated by a dashed line 2205. Then, by using these plots, the imageinformation complementing unit 16 e predicts a desired focus state plot2202 g′ at a signal accumulation time center 2202 g in the next frame2202 by using the adaptive filter or the linear prediction. As indicatedby a gray heavy line 2206, the focus lens 12 b is driven in a stepmanner during capturing the image in the next frame 2202. As describedabove, the readout of the image signals and each of the focus statedetection processing and the drive of the focus lens 12 b are temporallyshifted from each other, and accordingly it is possible to perform theoperations considering the electric power. The operations are repeatedfor each of frames 2203 and 2204, so that a continuous focused-imagecapturing can be performed.

FIG. 22B is an example of shielding light on the image pickup device 14between the continuous image captures by using a mechanical shutter 14a. In FIG. 22B, for each frame, the mechanical shutter 14 a is closed atthe time indicated by a line segment 2103, and thus the image pickupdevice 14 is shielded from an object light beam until the time indicatedby a next line segment 2104. Accordingly, in an interval indicated bymeshed lines 2105, image information from the object light beam cannotbe obtained. However, since sufficient image information can be obtainedbefore the light shield of the region is performed, image blurcorrection plots 2201 a′ to 2204 e′ can be obtained based on the imagesignals read during shielding the light. Then, based on these plots, theimage information complementing unit 16 e predicts a desired focus stateplot at the accumulation time center in the next frame by using theadaptive filter or the linear prediction. The focus lens 12 b is drivenin a step manner during the image capturing in the next frame 2202 asindicated by a gray heavy line 2206.

FIG. 22C illustrates a case in which allowance times 2207 and 2208 areprovided to detect and calculate the focus state between frames in eachphotography. The focus lens 12 b is driven during the allowance time.Based on plots 2201 a′ to 2201 e′ indicating the focus state, the imageinformation complementing unit 16 e predicts the desired focus stateplot 2202 g′ at the accumulation time center 2202 g in the next frame2202 by using the adaptive filter or the linear prediction. Then, duringthe allowance time 2207, the drive of the focus lens 12 b is completedat the time indicated by an arrow 2209 so as to be focused in the focusstate. Similarly, with respect to a frame 2203, the drive of the focuslens 12 b is completed at the time indicated by an arrow 2210. Asdescribed above, based on the plurality of image signals (focus state)obtained in one frame, the image information complementing unit 16 epredicts the target adjustment value of the image pickup unit relatingto subsequent frames (i.e., target adjustment value of the focus lens orthe blur correction lens), and accordingly a high-mobility imagecapturing operation can be performed.

As described above, in each embodiment, the image pickup unit can becontrolled by using the plurality of image signals in one frame, andaccordingly it is possible to perform the blur correction or the focuscorrection with high response. Furthermore, since the image informationis obtained by using different image signals in one frame, the noisesuperimposed on an image can be reduced compared with a conventionalmethod of reading the same image signals repeatedly in a nondestructivemanner.

The first embodiment describes the correction of the hand shake or thefocus shift that occurs in one frame by using the image blur correctionlens 12 a or the focus lens 12 b in the frame. The third embodimentdescribes the application of the correction to the still image. Thesixth embodiment describes the application to the continuous imagecapturing. However, each embodiment is not limited to the image blurcorrection or the focus correction, and it can be applied also to othercorrections such as a drive of the movable aperture stop 12 c. Forexample, a change of a brightness of an object obtained during capturinga moving image or a still image in a frame is obtained as imageinformation, and the movable aperture stop 12 c is driven in the framedepending on the image information. Alternatively, it is possible toimmediately correct the change of the brightness of the object obtainedin a previous frame during the continuous image capturing by moving themovable aperture stop.

As an example, referring to FIGS. 23A and 23B, the control of themovable aperture stop during the continuous image capturing will bedescribed. FIG. 23A is an explanatory diagram of the control of themovable aperture stop during the continuous image capturing. In thisembodiment, capturing an image of the object as illustrated in FIG. 9Cis considered. In this case, preferred photometric values are lightintensities in the regions 92 a to 92 e. Accordingly, an average lightintensity of the regions is obtained, and a change of the average lightintensity in each frame is set to a target drive value of the movableaperture stop.

In FIG. 23A, the average light intensity of regions 2202 a to 2202 e ina frame 2202 is read out. Subsequently, a comparison value result 2301of an average light intensity in all regions in the previous frame 2201and an average light intensity for each region and an average lightintensity of the regions 2202 a to 2202 e in the current frame arecalculated to obtain average light intensities 2202 a′ to 2202 e′ in allregions at the readout time in each region. The change of the obtainedlight intensity is extended to predict a light intensity 2203 g′ at thetime of an accumulation centroid 2203 g in the next frame 2203, and thenthe movable aperture stop is driven based on the prediction result. Thisoperation is repeated for each frame, and thus an appropriate exposureis always obtained for each frame in the continuous image capturing.

FIG. 23B is a flowchart of illustrating the control of the movableaperture stop illustrated in FIG. 23A, and for a simple explanation,elements which are not directly relevant to this embodiment are omitted.This flow starts when for example a user performs a photographingpreparation operation such as a half-press of a release button providedon the image pickup apparatus 100 (camera) and the operating unit 112outputs an instruction for the photographing preparation to the CPU 111,and it is finished when the instruction for the photographingpreparation is canceled.

First, at step S2301, the image information distribution detecting unit16 a selects a region in which photometry is to be performed as an imageinformation extraction range based on the image obtained by the imageprocessing unit 19 or the image information calculating unit 16.Selecting the region in which the photometry is to be performed means,for example, setting an object closest to the camera in FIG. 9C as amain object and setting an image region to capture the main object as aphotometry region. Subsequently, at step S2302, the signal readout unit15 reads image signals in each region. Subsequently, at step S2303, theCPU 111 detects an average light intensity in an entire region for eachof the regions of the read image signals. The average value in theentire region is used because there is a high possibility that the lightintensity greatly changes in a local region.

Subsequently, at step S2304, the signal readout unit 15 reads the nextregion. The signal readout unit 15 determines whether image signalsrequired for the image formation in one frame are read out. If the imagesignals are read out, the flow proceeds to step S2306. On the otherhand, if any region to be read in one frame remains, the flow returns tostep S2301, and the readout in the regions and the detection of theaverage light intensity are repeated.

At step S2306, the image information calculating unit 16 reads a lightintensity in each region read in a previous frame and an entirelyaverage light intensity in the previous frame image obtained byaveraging the read light intensities. Then, the image informationcalculating unit 16 obtains a light intensity in a current entire frameat the time of reading the region by the result of the previous frameand the average region of each region in the current frame. For example,if the average light intensity of a certain region in the current frameincreases by 10% with respect to the average light intensity of thecorresponding region in the previous frame, the entirely average lightintensity in the current frame at that time is set to a value obtainedby increasing the entirely average light intensity by 10% in theprevious frame.

Subsequently, at step S2307, the image information calculating unit 16predicts an entirely average light intensity 2203 g′ in the next framebased on a light intensity change curve 2205 obtained from entirelyaverage light intensities (2202 a′ to 2202 e′) at the time of readingsignals in each region obtained at step S2306. Subsequently, at stepS2308, the image information calculating unit 16 averages the lightintensities obtained in each region of the current frame and obtains theentirely average light intensity in the current frame image in order touse it for calculation of the light intensity in the next frame.Subsequently, at step S2309, the adjusting unit 17 drives a movableaperture stop 12 c via the drive unit 18 c based on the predicted lightintensity obtained at step S2307. Then, at step S2310, the processproceeds to the next frame.

Subsequently, at step S2311, the image processing unit 19 or the imageinformation calculating unit 16 determines whether a composition of animage is changed. If the composition is not changed, the flow returns tostep S2302, and it reads image signals also in the next frame with thecondition set at step S2301. On the other hand, if the composition ischanged, the flow returns to step S2301, and the setting of therange-finding frame and the setting of the readout order are performedagain.

While FIGS. 23A and 23B illustrates the example in which the movableaperture stop 12 c is intermittently driven for each frame, the movableaperture stop 12 c may be driven in real time based on an entire averagelight intensity at that time for each detection of the light intensityin each region. In this embodiment, the movable aperture stop 12 c ismechanically driven, and alternatively the brightness of an image can beadjusted without using the movable aperture stop 12 c. For example, byinputting brightness signals of an object to be sequentially output inone frame from the adjusting unit 17 to the image processing unit 19,the gain-up and the gain-down of the image can be performed with highresponse. As described above, by adopting the method of each embodimentto the system of performing the drive control of the blur correctionlens, the focus lens, and the movable aperture stop, or performing thegain correction on an image, the noise superimposed on the image can beeffectively reduced. Accordingly, a high-quality optical apparatus canbe achieved.

As described above, in each embodiment, an optical apparatus (imagepickup apparatus 100) includes a control apparatus including a signalreadout unit 15, an image information calculating unit 16, and anadjusting unit 17. The signal readout unit 15 reads out a frame image(an image in one frame) obtained from an image pickup device 14 whilethe frame image is divided into a plurality of different regions fromeach other. The image information calculating unit 16 calculates imageinformation based on an image signal of each of the plurality ofdifferent regions obtained from the signal readout unit 15. Theadjusting unit 17 determines a target adjustment value of an imagepickup unit based on image information during capturing the frame image.In each embodiment, at least one processor or circuit is configured toperform a function (operation) of at least one of the units.

In each embodiment, the image pickup apparatus 100 includes a camerabody 13 b including the image pickup device 14, and a lens barrel 13 aremovably attached to the camera body 13 b. Each embodiment is notlimited to the configuration in which all the signal readout unit 15,the image information calculating unit 16, and the adjusting unit 17that constitute the control apparatus are provided in the camera body 13b. At least one of the signal readout unit 15, the image informationcalculating unit 16, and the adjusting unit 17 may be provided in thelens barrel 13 a. For example, the optical apparatus is the lens barrel13 a (lens apparatus) removably attached to the image pickup apparatus100 including the image pickup device 14, and the optical apparatusincludes the signal readout unit 15, the image information calculatingunit 16, and the adjusting unit 17. The optical apparatus may be a lensbarrel removably attached to an image pickup apparatus including theimage pickup device 14 and the signal readout unit 15, and the opticalapparatus includes the image information calculating unit 16 and theadjusting unit 17. The optical apparatus may be a lens apparatusremovably attached to an image pickup apparatus including the imagepickup device 14, the signal readout unit 15, and the image informationcalculating unit 16, and the optical apparatus includes the adjustingunit 17.

According to each embodiment, a control apparatus, an image pickupapparatus, a control method, and a non-transitory computer-readablestorage medium which are capable of reducing a noise superimposed on animage when controlling an image pickup unit based on a plurality ofimage signals in a frame can be provided.

Other Embodiments

Embodiment (s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-094085, filed on May 1, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A control apparatus comprising: a signal readoutunit configured to read out a frame image obtained from an image pickupdevice while the frame image is divided into a plurality of differentregions; an image information calculating unit configured to calculateimage information based on an image signal of each of the plurality ofdifferent regions obtained from the signal readout unit; and anadjusting unit configured to determine a target adjustment value of animage pickup unit including an image pickup optical system and the imagepickup device based on the image information during capturing the frameimage, wherein at least one processor or circuit is configured toperform a function of at least one of the units.
 2. The controlapparatus according to claim 1, further comprising a drive unitconfigured to drive the image pickup unit based on a signal from theadjusting unit during capturing the frame image.
 3. The controlapparatus according to claim 1, wherein the image informationcalculating unit includes: an image information distribution detectingunit configured to determine an image information extraction range fromwhich the image information is extracted and detect a distribution ofthe determined image information extraction range, a feature pointcoordinate calculating unit configured to calculate a coordinate of afirst image information extraction range determined at the time ofreading out a first image signal of the image signals, a feature pointcoordinate estimating unit configured to estimate a coordinate of asecond image information extraction range different from the first imageinformation extraction range based on the coordinate of the first imageinformation extraction range calculated by the feature point coordinatecalculating unit, and an image information comparing unit configured tocompare the coordinate of the second image information extraction rangeestimated by the feature point coordinate estimating unit with thecoordinate of the first image information extraction range calculated bythe feature point coordinate calculating unit determined after a passageof time from a readout time of the first image signal.
 4. The controlapparatus according to claim 3, wherein the signal readout unit isconfigured to preferentially read out a region including the imageinformation extraction range in the plurality of different regions ofthe frame image.
 5. The control apparatus according to claim 3, whereinthe signal readout unit is configured to perform decimating readout of aregion other than the region including the image information extractionrange in the plurality of different regions of the frame image.
 6. Thecontrol apparatus according to claim 3, wherein the signal readout unitis configured to change a readout order based on the distribution of theimage information extraction range.
 7. The control apparatus accordingto claim 3, wherein the image information distribution detecting unit isconfigured to average coordinates of corresponding image informationextraction ranges in a plurality of frame images to detect thedistribution of the image information extraction range.
 8. The controlapparatus according to claim 3, wherein the image informationdistribution detecting unit is configured to detect the distribution ofthe image information extraction range based on a change of coordinatesof corresponding image information extraction ranges in a plurality offrame images.
 9. The control apparatus according to claim 3, wherein thesignal readout unit is configured to read out a region including theimage information extraction range at a constant interval for the frameimage based on the distribution of the image information extractionrange.
 10. The control apparatus according to claim 9, wherein thesignal readout unit is configured to alternately read out a region whichincludes the image information extraction range and a region which doesnot include the image information extraction range in the plurality ofdifferent regions.
 11. The control apparatus according to claim 10,wherein the signal readout unit is configured to divide the plurality ofdifferent regions such that the number of regions each of which includesthe image information extraction range and the number of regions each ofwhich does not include the image information extraction range are thesame each other.
 12. The control apparatus according to claim 3, whereinthe feature point coordinate calculating unit is configured to comparethe plurality of image information extraction ranges of a plurality offrames for each corresponding image information extraction range toobtain a plurality of comparison waveforms, and calculate the targetadjustment value based on a relationship of the plurality of comparisonwaveforms.
 13. The control apparatus according to claim 12, wherein theimage information distribution detecting unit is configured to determinea relative coordinate of image information extraction ranges differentfrom each other in a frame based on the plurality of comparisonwaveforms.
 14. The control apparatus according to claim 13, wherein thefeature point coordinate calculating unit is configured to calculate thetarget adjustment value in a frame relating to a next frame based on arelationship of a plurality of image information extraction ranges in aframe determined by the image information distribution detecting unit.15. The control apparatus according to claim 3, wherein the imageinformation distribution detecting unit is configured to segmentalizethe plurality of different regions before capturing a still image. 16.The control apparatus according to claim 1, wherein the signal readoutunit is configured to change a time of a signal readout depending on asignal accumulation time required for capturing a still image.
 17. Thecontrol apparatus according to claim 3, further comprising an imageprocessing unit configured to synthesize a first frame that is used toobtain the target adjustment value with a second frame different fromthe first frame to generate a still image.
 18. The control apparatusaccording to claim 17, wherein the image information distributiondetecting unit associates a readout time of a plurality of framesadjacent to each other with a signal accumulation time.
 19. The controlapparatus according to claim 17, wherein the signal readout unit delaysa signal readout time compared with a setting time in an accumulationstart scan for a frame.
 20. The control apparatus according to claim 19,wherein the signal readout unit is configured to set the signal readouttime to be earlier than the setting time in a frame next to the frame inwhich the signal readout time is delayed compared with the setting time.21. The control apparatus according to claim 17, further comprising anaccumulation control unit configured to control a start of anaccumulation of a frame which is to be initially captured in a pluralityof frames synthesized by the image processing unit.
 22. The controlapparatus according to claim 17, further comprising a readout controlunit configured to control a signal readout of a frame which is to befinally captured in a plurality of frames synthesized by the imageprocessing unit.
 23. The control apparatus according to claim 22,wherein the readout control unit is configured to perform an image blurcorrection by using an image signal read out while light to be incidenton the image pickup device is shielded.
 24. The control apparatusaccording to claim 1, wherein the image information calculating unitincludes an image information complementing unit configured tocomplement image information of an image information lacked portionbased on the image information.
 25. The control apparatus according toclaim 1, wherein the image information calculating unit includes animage information complementing unit configured to perform a gainadjustment of the image signal depending on an accumulation timedifference in the frame image.
 26. The control apparatus according toclaim 1, wherein the image information calculating unit includes animage information complementing unit configured to predict the targetadjustment value relating to subsequent frames based on a plurality ofimage signals in a frame.
 27. An optical apparatus comprising: an imagepickup device; a signal readout unit configured to read out a frameimage obtained from the image pickup device while the frame image isdivided into a plurality of different regions; an image informationcalculating unit configured to calculate image information based on animage signal of each of the plurality of different regions obtained fromthe signal readout unit; and an adjusting unit configured to determine atarget adjustment value of an image pickup unit including an imagepickup optical system and the image pickup device based on the imageinformation during capturing the frame image, wherein at least oneprocessor or circuit is configured to perform a function of at least oneof the units.
 28. An optical apparatus removably attached to an imagepickup apparatus including an image pickup device, the optical apparatuscomprising: a signal readout unit configured to read out a frame imageobtained from the image pickup device while the frame image is dividedinto a plurality of different regions; an image information calculatingunit configured to calculate image information based on an image signalof each of the plurality of different regions obtained from the signalreadout unit; and an adjusting unit configured to determine a targetadjustment value of an image pickup unit including an image pickupoptical system and the image pickup device based on the imageinformation during capturing the frame image, wherein at least oneprocessor or circuit is configured to perform a function of at least oneof the units.
 29. An optical apparatus removably attached to an imagepickup apparatus including an image pickup device and a signal readoutunit configured to read out a frame image obtained from the image pickupdevice while the frame image is divided into a plurality of differentregions, the optical apparatus comprising: an image informationcalculating unit configured to calculate image information based on animage signal of each of the plurality of different regions obtained fromthe signal readout unit; and an adjusting unit configured to determine atarget adjustment value of an image pickup unit including an imagepickup optical system and the image pickup device based on the imageinformation during capturing the frame image, wherein at least oneprocessor or circuit is configured to perform a function of at least oneof the units.
 30. An optical apparatus removably attached to an imagepickup apparatus including an image pickup device, a signal readout unitconfigured to read out a frame image obtained from the image pickupdevice while the frame image is divided into a plurality of differentregions, and an image information calculating unit configured tocalculate image information based on an image signal of each of theplurality of different regions obtained from the signal readout unit,the optical apparatus comprising: an adjusting unit configured todetermine a target adjustment value of an image pickup unit including animage pickup optical system and the image pickup device based on theimage information during capturing the frame image, wherein at least oneprocessor or circuit is configured to perform a function of at least oneof the units.
 31. A control method comprising the steps of: reading outa frame image obtained from an image pickup device while the frame imageis divided into a plurality of different regions; calculating imageinformation based on an image signal of each of the plurality ofdifferent regions; and determining a target adjustment value of an imagepickup unit based on the image information during capturing the frameimage.
 32. A non-transitory computer-readable storage medium storing aprogram which causes a computer to execute a process comprising thesteps of: reading out a frame image obtained from an image pickup devicewhile the frame image is divided into a plurality of different regions;calculating image information based on an image signal of each of theplurality of different regions; and determining a target adjustmentvalue of an image pickup unit based on the image information duringcapturing the frame image.